Article Text

PDF

Initial validation of the International Crowding Measure in Emergency Departments (ICMED) to measure emergency department crowding
  1. Adrian Boyle1,
  2. James Coleman2,
  3. Yasmin Sultan3,
  4. Vijayasankar Dhakshinamoorthy4,
  5. Jacqueline O'Keeffe5,
  6. Pramin Raut2,
  7. Kathleen Beniuk6
  1. 1Emergency Department, Addenbrooke's Hospital Cambridge University Hospitals NHS Foundation Trust, Cambridge, Cambridgeshire, UK
  2. 2Clinical School, Cambridge University, Cambridge, Cambridgeshire, UK
  3. 3Emergency Department, Addenbrooke's Hospital, Cambridge University Hospitals NHS Foundation Trust, Cambridge, Cambridgeshire, UK
  4. 4Emergency Department, Peterborough and Stamford Hospitals NHS Foundation Hospitals Trust, Peterborough, UK
  5. 5Emergency Department, Hinchingbrooke Healthcare NHS Trust, Huntingdon, UK
  6. 6Engineering Design Centre, Cambridge University, Cambridge, Cambridgeshire, UK
  1. Correspondence to Dr Adrian Boyle, Emergency Department, Addenbrooke's Hospital Cambridge University Hospitals NHS Foundation Trust, Cambridge, Cambridgeshire CB2 2QQ, UK; Adrian.boyle{at}addenbrookes.nhs.uk

Abstract

Introduction Emergency department (ED) crowding is recognised as a major public health problem. While there is agreement that ED crowding harms patients, there is less agreement about the best way to measure ED crowding. We have previously derived an eight-point measure of ED crowding by a formal consensus process, the International Crowding Measure in Emergency Departments (ICMED). We aimed to test the feasibility of collecting this measure in real time and to partially validate this measure.

Methods We conducted a cross-sectional study in four EDs in England. We conducted independent observations of the measure and compared these with senior clinician's perceptions of crowding and safety.

Results We obtained 84 measurements spread evenly across the four EDs. The measure was feasible to collect in real time except for the ‘Left Before Being Seen’ variable. Increasing numbers of violations of the measure were associated with increasing clinician concerns. The area under the receiver operating characteristic curve was 0.80 (95% CI 0.72 to 0.90) for predicting crowding and 0.74 (95% CI 0.60 to 0.89) for predicting danger. The optimal number of violations for predicting crowding was three, with a sensitivity of 91.2 (95% CI 85.1 to 97.2) and a specificity of 100.0 (92.9–100). The measure predicted clinician concerns better than individual variables such as occupancy.

Discussion The ICMED can easily be collected in multiple EDs with different information technology systems. The ICMED seems to predict clinician's concerns about crowding and safety well, but future work is required to validate this before it can be advocated for widespread use.

  • emergency care systems, efficiency
  • emergency department management
  • systems
  • management, emergency department management

Statistics from Altmetric.com

Introduction

Emergency department (ED) crowding is increasingly recognised as a global public health problem. Crowding impacts on both patients and staff. Delays to diagnosis and treatment are well described.1–10 Privacy and dignity are compromised and mortality is greater in patients who are admitted through crowded EDs.11 ,12 Crowding decreases staff and patient satisfaction and increases burnout.13 Recruitment and retention of ED staff is harmed, training suffers.14–16

While there is widespread acceptance that ED crowding is important, there is little consensus on the best way to measure this.17 Hwang and Concato stress the need to have some conceptual discipline because ‘separating factors that cause overcrowding from the phenomenon itself can assist in the development of interventions that ease overcrowding’.18 Measuring crowding is important because measurement allows the development and evaluation of interventions.19 ,20 Many studies use the need for ambulance diversion as a proxy measure for ED crowding, though this practice is not widespread and depends on factors other than ED crowding.21 Occupancy is also used as a measure, as it is easy to collect.22 The proportion of patients leaving being seen is also easy to measure, but does not adequately measure the problem.23 The National Emergency Department Overcrowding Score (NEDOCS) measure has been developed and is used inconsistently.24 NEDOCS measure correlates well with the proportion of patients leaving before being seen. The Emergency Department Work Index (EDWIN) is limited by relying on a five-point triage scale, which is less relevant to many EDs which use a three-point scale.25

Our research group has previously developed, by a formal consensus process involving 40 emergency physicians across the world, a measure of ED crowding.26 This is an eight-point measure, called the International Crowding Measure in Emergency Departments (ICMED), shown in table 1.

Table 1

International Crowding Measure in Emergency Departments (ICMED)

Most measures of crowding have not been validated. Validation of a crowding measure needs to demonstrate several forms of validity. Demonstrating face validity is important to ensure uptake among clinicians. The aims of this study were to assess the feasibility of recording the components of this scale, partially validate the ICMED, determine a cut-off level at which we would identify an ED as crowded or not and finally, obtain data to plan a sample size for a definitive validation study.

Methods

We performed a series of cross-sectional observations across four regional hospitals in the East of England; these serve mixed urban and rural areas in autumn 2012. One was a very large teaching hospital, one was a large district general hospital and the remaining two were small district general hospitals. One of us, JC, recorded the status of the ED using seven of the eight measures in real time every hour. The ‘left before treatment’ variable cannot easily be recorded in real time, but is best reported historically. At the same time, the consultant in charge of the ED was asked to record his/her opinion of how crowded and dangerous the ED was on a 10 cm Visual Analogue Scale ranging from ‘0’ (not at all crowded) to 10 (extremely crowded).

The clinician was blinded to the scores recording violations of the crowding scales, but fully aware of the pressure that their department was under. We purposively sampled mainly through the afternoon and the evening and a mix of weekdays and weekends, as when historical data indicated the demands for service were greatest and showed the greatest changes.

We were advised by our Local Research and Ethics Committee that formal ethical approval was unnecessary as this was a service evaluation and no patient contact was proposed. We analysed the data in STATA V.12. We report the mean crowding and danger scores at each level of violation. Second, we transformed the clinician perception scores into a binary score above and below five on the Visual Analogue Scale. We regarded, based on consensus, scores of greater than five indicated that an ED was crowded and that a ‘danger score’ of greater than five indicated a dangerous department. We analysed the data as a diagnostic test, with the component variables and increasing numbers of violations of the ICMED being compared with the gold standard of clinician perception of crowding and safety. We used the ROCTAB function of STATA to generate a receiver operating characteristic (ROC) curve.27 We conducted a one afternoon pilot to test the feasibility of the questionnaire. The results from this pilot were not included in the final analysis. There were no sample size data to guide this study.

Results

We conducted 84 h of observations in four different EDs. There were no missing data items. Seven consultant emergency physicians provided scores on the two Visual Analogue Scales.

Consultant perceptions of crowding and danger were well correlated, Spearman's rho=0.60 and p<0.001, though danger tended to attract a lower score. The consultants graded their departments as more than five on the crowding Visual Analogue Scale 40% of the time and as more than five on the danger Visual Analogue Scale 15% of the time. Figure 1 shows that as the number of violations increased, the mean perceptions of crowding and danger increased. Table 2 shows the mean perceptions of concern.

Table 2

Mean scores of crowding and danger

Figure 1

Mean perceptions of danger and crowding against increasing numbers of violations of the International Crowding Measure in Emergency Departments (ICMED).

Table 3 shows the diagnostic performance of the ICMED against clinician's perception of crowding. Numbers of violations of the ICMED tended to perform better than individual items. Three violations of the ICMED seemed to have the optimal performance in predicting clinician's concerns about crowding. Table 4 shows the diagnostic performance of the ICMED against clinician's perception of danger. The ICMED performed less well at predicting clinician's concerns about danger, in part because there were fewer times where an ED was felt to be dangerous.

Table 3

Performance of the ICMED against clinician perception of crowding

Table 4

Performance of the ICMED against clinician perception of danger

The ICMED had an area under the ROC curve (AuROC) of 0.80 (95% CI 0.72 to 0.90) for predicting crowding and 0.74 (95% CI 0.60 to 0.89) for danger, respectively. The measure performed less well at predicting clinician's perceptions of danger, in part because there were not very many data points that indicated a high perception of danger.

Discussion

We have evaluated our derived measure of ED crowding. We have demonstrated that the ICMED can easily be collected in four diverse EDs that all have different information systems and that seven measures can be collected in real time. We have demonstrated that increasing numbers of violations are associated with increased clinician perceptions of crowding and danger.

We have also demonstrated that the combination of variables predicts a clinician's concerns better than individual variables. This is important as many studies use occupancy as their sole measure of crowding. Our results suggest that the optimal number of violations that predict clinician's perceptions of crowding is three. The ‘Left Before Being Seen’ variable was not feasible to collect in real time, but we suggest that this has value in operational performance management as it is easily collected and has high face validity.

The ICMED predicted clinician's perception of crowding well, with an AuROC of 0.80. There are several other measures of ED crowding reported in the literature.20 ,18 This AuROC is comparable with the highest reported values of NEDOCS (0.92) and EDWIN (0.84), the best studied measures of ED crowding.28 When we used the three violations as a predictor of ED crowding, ICMED exhibited higher sensitivity (91.2) and specificity (100) than other measures of ED crowding. There are also concerns whether the NEDOCS measure works well outside the USA and whether it is too complex.24 ,18

Limitations

There are some important limitations to our work. We took a convenience sample. Our sampling frame contained very few times when an ED was under extreme pressure. We did not sample between 22:00 and 08:00. It is possible that our results would differ overnight, as the causes of adverse incidents may be very different after midnight. Our anecdotal experience is that crowding mainly occurs in the afternoon and evening. Our gold standard of clinician perceptions of crowding and danger is not validated, but there is no widely accepted, properly validated measure. It has been suggested that clinician perception of crowding alone could be used, however the validity and reliability of this is uncertain, we think the ICMED would perform better as it addresses the components of crowding and is more objective. Some North American studies have evaluated crowding measures against the need for ambulance diversion; this response to ED crowding is rarely used in the UK. There is also US evidence that many EDs are very crowded, but do not use ambulance diversion.29 The physicians were not blinded to the variables that we were collecting, as this information is used to run an ED. It would have been unfeasible and unethical to blind the physicians from information that they routinely use. The measure was developed by an international consensus, but it is not clear whether our results would have been similar if we had conducted the study internationally.

Future work

Our results should be regarded as exploratory and should not be used in routine ED operational management yet. A larger validation study with ‘harder’ end points, such as inpatient mortality, length of hospital stay and clinical incidents, is required before this measure can be adopted for routine health management. Our measure should also be compared with other widely used measures of ED crowding, the most commonly used measures such as the NEDOCS and the EDWIN. Our measure should also be compared with patient satisfaction scores.

Conclusions

The ICMED is feasible to collect in UK EDs. A combination of violations, probably three, predicts clinician concerns better than individual violations. Future work is required to validate this before this can be used.

References

View Abstract

Footnotes

  • Contributors AB and JC conceived the study. JC collected the data. AB, JC and PR analysed the data and wrote the report. YS, DK and JOK facilitated the study and supported data collection. All authors have read the final report.

  • Competing interests None.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles