You are here

  • Home
  • Archive
  • Volume 32, Issue 2
  • Initial validation of the International Crowding Measure in Emergency Departments (ICMED) to measure emergency department crowding

Article Text

Article menu

Download PDFPDF

Original article
Initial validation of the International Crowding Measure in Emergency Departments (ICMED) to measure emergency department crowding
Free
  1. Adrian Boyle1,
  2. James Coleman2,
  3. Yasmin Sultan3,
  4. Vijayasankar Dhakshinamoorthy4,
  5. Jacqueline O'Keeffe5,
  6. Pramin Raut2,
  7. Kathleen Beniuk6
  1. 1Emergency Department, Addenbrooke's Hospital Cambridge University Hospitals NHS Foundation Trust, Cambridge, Cambridgeshire, UK
  2. 2Clinical School, Cambridge University, Cambridge, Cambridgeshire, UK
  3. 3Emergency Department, Addenbrooke's Hospital, Cambridge University Hospitals NHS Foundation Trust, Cambridge, Cambridgeshire, UK
  4. 4Emergency Department, Peterborough and Stamford Hospitals NHS Foundation Hospitals Trust, Peterborough, UK
  5. 5Emergency Department, Hinchingbrooke Healthcare NHS Trust, Huntingdon, UK
  6. 6Engineering Design Centre, Cambridge University, Cambridge, Cambridgeshire, UK
  1. Correspondence to Dr Adrian Boyle, Emergency Department, Addenbrooke's Hospital Cambridge University Hospitals NHS Foundation Trust, Cambridge, Cambridgeshire CB2 2QQ, UK; Adrian.boyle{at}addenbrookes.nhs.uk

Abstract

Introduction Emergency department (ED) crowding is recognised as a major public health problem. While there is agreement that ED crowding harms patients, there is less agreement about the best way to measure ED crowding. We have previously derived an eight-point measure of ED crowding by a formal consensus process, the International Crowding Measure in Emergency Departments (ICMED). We aimed to test the feasibility of collecting this measure in real time and to partially validate this measure.

Methods We conducted a cross-sectional study in four EDs in England. We conducted independent observations of the measure and compared these with senior clinician's perceptions of crowding and safety.

Results We obtained 84 measurements spread evenly across the four EDs. The measure was feasible to collect in real time except for the ‘Left Before Being Seen’ variable. Increasing numbers of violations of the measure were associated with increasing clinician concerns. The area under the receiver operating characteristic curve was 0.80 (95% CI 0.72 to 0.90) for predicting crowding and 0.74 (95% CI 0.60 to 0.89) for predicting danger. The optimal number of violations for predicting crowding was three, with a sensitivity of 91.2 (95% CI 85.1 to 97.2) and a specificity of 100.0 (92.9–100). The measure predicted clinician concerns better than individual variables such as occupancy.

Discussion The ICMED can easily be collected in multiple EDs with different information technology systems. The ICMED seems to predict clinician's concerns about crowding and safety well, but future work is required to validate this before it can be advocated for widespread use.

  • emergency care systems, efficiency
  • emergency department management
  • systems
  • management, emergency department management

https://doi.org/10.1136/emermed-2013-202849

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Emergency department (ED) crowding is increasingly recognised as a global public health problem. Crowding impacts on both patients and staff. Delays to diagnosis and treatment are well described.1–10 Privacy and dignity are compromised and mortality is greater in patients who are admitted through crowded EDs.11 ,12 Crowding decreases staff and patient satisfaction and increases burnout.13 Recruitment and retention of ED staff is harmed, training suffers.14–16

    While there is widespread acceptance that ED crowding is important, there is little consensus on the best way to measure this.17 Hwang and Concato stress the need to have some conceptual discipline because ‘separating factors that cause overcrowding from the phenomenon itself can assist in the development of interventions that ease overcrowding’.18 Measuring crowding is important because measurement allows the development and evaluation of interventions.19 ,20 Many studies use the need for ambulance diversion as a proxy measure for ED crowding, though this practice is not widespread and depends on factors other than ED crowding.21 Occupancy is also used as a measure, as it is easy to collect.22 The proportion of patients leaving being seen is also easy to measure, but does not adequately measure the problem.23 The National Emergency Department Overcrowding Score (NEDOCS) measure has been developed and is used inconsistently.24 NEDOCS measure correlates well with the proportion of patients leaving before being seen. The Emergency Department Work Index (EDWIN) is limited by relying on a five-point triage scale, which is less relevant to many EDs which use a three-point scale.25

    Our research group has previously developed, by a formal consensus process involving 40 emergency physicians across the world, a measure of ED crowding.26 This is an eight-point measure, called the International Crowding Measure in Emergency Departments (ICMED), shown in table 1.

    View this table:
    Table 1

    International Crowding Measure in Emergency Departments (ICMED)

    Most measures of crowding have not been validated. Validation of a crowding measure needs to demonstrate several forms of validity. Demonstrating face validity is important to ensure uptake among clinicians. The aims of this study were to assess the feasibility of recording the components of this scale, partially validate the ICMED, determine a cut-off level at which we would identify an ED as crowded or not and finally, obtain data to plan a sample size for a definitive validation study.

    Methods

    We performed a series of cross-sectional observations across four regional hospitals in the East of England; these serve mixed urban and rural areas in autumn 2012. One was a very large teaching hospital, one was a large district general hospital and the remaining two were small district general hospitals. One of us, JC, recorded the status of the ED using seven of the eight measures in real time every hour. The ‘left before treatment’ variable cannot easily be recorded in real time, but is best reported historically. At the same time, the consultant in charge of the ED was asked to record his/her opinion of how crowded and dangerous the ED was on a 10 cm Visual Analogue Scale ranging from ‘0’ (not at all crowded) to 10 (extremely crowded).

    The clinician was blinded to the scores recording violations of the crowding scales, but fully aware of the pressure that their department was under. We purposively sampled mainly through the afternoon and the evening and a mix of weekdays and weekends, as when historical data indicated the demands for service were greatest and showed the greatest changes.

    We were advised by our Local Research and Ethics Committee that formal ethical approval was unnecessary as this was a service evaluation and no patient contact was proposed. We analysed the data in STATA V.12. We report the mean crowding and danger scores at each level of violation. Second, we transformed the clinician perception scores into a binary score above and below five on the Visual Analogue Scale. We regarded, based on consensus, scores of greater than five indicated that an ED was crowded and that a ‘danger score’ of greater than five indicated a dangerous department. We analysed the data as a diagnostic test, with the component variables and increasing numbers of violations of the ICMED being compared with the gold standard of clinician perception of crowding and safety. We used the ROCTAB function of STATA to generate a receiver operating characteristic (ROC) curve.27 We conducted a one afternoon pilot to test the feasibility of the questionnaire. The results from this pilot were not included in the final analysis. There were no sample size data to guide this study.

    Results

    We conducted 84 h of observations in four different EDs. There were no missing data items. Seven consultant emergency physicians provided scores on the two Visual Analogue Scales.

    Consultant perceptions of crowding and danger were well correlated, Spearman's rho=0.60 and p<0.001, though danger tended to attract a lower score. The consultants graded their departments as more than five on the crowding Visual Analogue Scale 40% of the time and as more than five on the danger Visual Analogue Scale 15% of the time. Figure 1 shows that as the number of violations increased, the mean perceptions of crowding and danger increased. Table 2 shows the mean perceptions of concern.

    View this table:
    Table 2

    Mean scores of crowding and danger

    Figure 1
    Figure 1

    Mean perceptions of danger and crowding against increasing numbers of violations of the International Crowding Measure in Emergency Departments (ICMED).

    Table 3 shows the diagnostic performance of the ICMED against clinician's perception of crowding. Numbers of violations of the ICMED tended to perform better than individual items. Three violations of the ICMED seemed to have the optimal performance in predicting clinician's concerns about crowding. Table 4 shows the diagnostic performance of the ICMED against clinician's perception of danger. The ICMED performed less well at predicting clinician's concerns about danger, in part because there were fewer times where an ED was felt to be dangerous.

    View this table:
    Table 3

    Performance of the ICMED against clinician perception of crowding

    View this table:
    Table 4

    Performance of the ICMED against clinician perception of danger

    The ICMED had an area under the ROC curve (AuROC) of 0.80 (95% CI 0.72 to 0.90) for predicting crowding and 0.74 (95% CI 0.60 to 0.89) for danger, respectively. The measure performed less well at predicting clinician's perceptions of danger, in part because there were not very many data points that indicated a high perception of danger.

    Discussion

    We have evaluated our derived measure of ED crowding. We have demonstrated that the ICMED can easily be collected in four diverse EDs that all have different information systems and that seven measures can be collected in real time. We have demonstrated that increasing numbers of violations are associated with increased clinician perceptions of crowding and danger.

    We have also demonstrated that the combination of variables predicts a clinician's concerns better than individual variables. This is important as many studies use occupancy as their sole measure of crowding. Our results suggest that the optimal number of violations that predict clinician's perceptions of crowding is three. The ‘Left Before Being Seen’ variable was not feasible to collect in real time, but we suggest that this has value in operational performance management as it is easily collected and has high face validity.

    The ICMED predicted clinician's perception of crowding well, with an AuROC of 0.80. There are several other measures of ED crowding reported in the literature.20 ,18 This AuROC is comparable with the highest reported values of NEDOCS (0.92) and EDWIN (0.84), the best studied measures of ED crowding.28 When we used the three violations as a predictor of ED crowding, ICMED exhibited higher sensitivity (91.2) and specificity (100) than other measures of ED crowding. There are also concerns whether the NEDOCS measure works well outside the USA and whether it is too complex.24 ,18

    Limitations

    There are some important limitations to our work. We took a convenience sample. Our sampling frame contained very few times when an ED was under extreme pressure. We did not sample between 22:00 and 08:00. It is possible that our results would differ overnight, as the causes of adverse incidents may be very different after midnight. Our anecdotal experience is that crowding mainly occurs in the afternoon and evening. Our gold standard of clinician perceptions of crowding and danger is not validated, but there is no widely accepted, properly validated measure. It has been suggested that clinician perception of crowding alone could be used, however the validity and reliability of this is uncertain, we think the ICMED would perform better as it addresses the components of crowding and is more objective. Some North American studies have evaluated crowding measures against the need for ambulance diversion; this response to ED crowding is rarely used in the UK. There is also US evidence that many EDs are very crowded, but do not use ambulance diversion.29 The physicians were not blinded to the variables that we were collecting, as this information is used to run an ED. It would have been unfeasible and unethical to blind the physicians from information that they routinely use. The measure was developed by an international consensus, but it is not clear whether our results would have been similar if we had conducted the study internationally.

    Future work

    Our results should be regarded as exploratory and should not be used in routine ED operational management yet. A larger validation study with ‘harder’ end points, such as inpatient mortality, length of hospital stay and clinical incidents, is required before this measure can be adopted for routine health management. Our measure should also be compared with other widely used measures of ED crowding, the most commonly used measures such as the NEDOCS and the EDWIN. Our measure should also be compared with patient satisfaction scores.

    Conclusions

    The ICMED is feasible to collect in UK EDs. A combination of violations, probably three, predicts clinician concerns better than individual violations. Future work is required to validate this before this can be used.

    References

    1. ↵
      1. Jo S,
      2. Kim K,
      3. Lee JH,
      4. et al
      . Emergency department crowding is associated with 28-day mortality in community-acquired pneumonia patients. J Infect 2012;64:268–75.
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      1. Kennebeck SS,
      2. Timm NL,
      3. Kurowski EM,
      4. et al
      . The association of emergency department crowding and time to antibiotics in febrile neonates. Acad Emerg Med 2011;18:1380–5.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. Sills MR,
      2. Fairclough DL,
      3. Ranade D,
      4. et al
      . Emergency department crowding is associated with decreased quality of analgesia delivery for children with pain related to acute, isolated, long-bone fractures. Acad Emerg Med 2011;18:1330–8.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Epstein SK,
      2. Huckins DS,
      3. Liu SW,
      4. et al
      . Emergency department crowding and risk of preventable medical errors. Intern Emerg Med 2012;7:173–80.
      OpenUrlPubMed
    5. ↵
      1. Sills MR,
      2. Fairclough D,
      3. Ranade D,
      4. et al
      . Emergency department crowding is associated with decreased quality of care for children. Pediatr Emerg Care 2011;27:837–45.
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Sills MR,
      2. Fairclough D,
      3. Ranade D,
      4. et al
      . Emergency department crowding is associated with decreased quality of care for children with acute asthma. Ann Emerg Med 2011;57:191–200.
      OpenUrlCrossRefPubMed
    7. ↵
      1. Pines JM,
      2. Pollack CV Jr..,
      3. Diercks DB,
      4. et al
      . The association between emergency department crowding and adverse cardiovascular outcomes in patients with chest pain. Acad Emerg Med 2009;16:617–25.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Moskop JC,
      2. Sklar DP,
      3. Geiderman JM,
      4. et al
      . Emergency department crowding, part 1–concept, causes, and moral consequences. Ann Emerg Med 2009;53:605–11.
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Bernstein SL,
      2. Aronsky D,
      3. Duseja R,
      4. et al
      . The effect of emergency department crowding on clinically oriented outcomes. Acad Emerg Med 2009;16: 1–10.
      OpenUrlCrossRefPubMedWeb of Science
    10. ↵
      1. Hoot NR,
      2. Aronsky D
      . Systematic review of emergency department crowding: causes, effects, and solutions. Ann Emerg Med 2008;52:126–36.
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Plunkett PK,
      2. Byrne DG,
      3. Breslin T,
      4. et al
      . Increasing wait times predict increasing mortality for emergency medical admissions. Eur J Emerg Med 2011;18: 192–6.
      OpenUrlCrossRefPubMed
    12. ↵
      1. Sun BC,
      2. Hsia RY,
      3. Weiss RE,
      4. et al
      . Effect of emergency department crowding on outcomes of admitted patients. Ann Emerg Med 2013;61:605–11.
      OpenUrlCrossRefPubMed
    13. ↵
      1. Tekwani KL,
      2. Kerem Y,
      3. Mistry CD,
      4. et al
      . Emergency department crowding is associated with reduced satisfaction scores in patients discharged from the emergency department. West J Emerg Med 2013;14:11–15.
      OpenUrlCrossRefPubMed
    14. ↵
      1. Mahler SA,
      2. McCartney JR,
      3. Swoboda TK,
      4. et al
      . The impact of emergency department overcrowding on resident education. J Emerg Med 2012;42:69–73.
      OpenUrlCrossRefPubMed
    15. ↵
      1. Popa F,
      2. Raed A,
      3. Purcarea VL,
      4. et al
      . Occupational burnout levels in emergency medicine–a nationwide study and analysis. J Med Life 2010;3:207–15.
      OpenUrlPubMed
    16. ↵
      1. Sikka R,
      2. Mehta S,
      3. Kaucky C,
      4. et al
      . ED crowding is associated with an increased time to pneumonia treatment. Am J Emerg Med 2010;28:809–12.
      OpenUrlCrossRefPubMed
    17. ↵
      1. Higginson I
      . Emergency department crowding. Emerg Med J 2012;29:437–43.
      OpenUrlAbstract/FREE Full Text
    18. ↵
      1. Hwang U,
      2. McCarthy ML,
      3. Aronsky D,
      4. et al
      . Measures of crowding in the emergency department: a systematic review. Acad Emerg Med 2011;18:527–38.
      OpenUrlCrossRefPubMed
    19. ↵
      1. Morris ZS,
      2. Boyle A,
      3. Beniuk K,
      4. et al
      . Emergency department crowding: towards an agenda for evidence-based intervention. Emerg Med J 2012;29:460–6.
      OpenUrlAbstract/FREE Full Text
    20. ↵
      1. Boyle A,
      2. Beniuk K,
      3. Higginson I,
      4. et al
      . Emergency department crowding: time for interventions and policy evaluations. Emerg Med Int 2012;2012:1–8.
      OpenUrl
    21. ↵
      1. Shenoi RP,
      2. Ma L,
      3. Jones J
      ,et al. Ambulance diversion as a proxy for emergency department crowding: the effect on pediatric mortality in a metropolitan area. Acad Emerg Med 2009;16:116–23.
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. McCarthy ML,
      2. Aronsky D,
      3. Jones ID,
      4. et al
      . The emergency department occupancy rate: a simple measure of emergency department crowding. Ann Emerg Med 2008;51:15–24, 24.
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. Pines JM
      . The left-without-being-seen rate: an imperfect measure of emergency department crowding. Acad Emerg Med 2006;13:807–8.
      OpenUrlCrossRefPubMed
    24. ↵
      1. Raj K,
      2. Baker K,
      3. Brierley S,
      4. et al
      . National Emergency Department Overcrowding Study tool is not useful in an Australian emergency department. Emerg Med Australas 2006;18:282–8.
      OpenUrlCrossRefPubMed
    25. ↵
      1. Bernstein SL,
      2. Verghese V,
      3. Leung W,
      4. et al
      . Development and validation of a new index to measure emergency department crowding. Acad Emerg Med 2003;10:938–42.
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      1. Beniuk K,
      2. Boyle AA,
      3. Clarkson PJ
      . Emergency department crowding: prioritising quantified crowding measures using a Delphi study. Emerg Med J 2012;29:868–71.
      OpenUrlAbstract/FREE Full Text
    27. ↵
      1. Pepe MS
      . Estimation and comparison of receiver operating characteristic curves. The Stata Journal 2009;9:1–16.
      OpenUrlPubMedWeb of Science
    28. ↵
      1. Jones SS,
      2. Allen TL,
      3. Flottemesch TJ,
      4. et al
      . An independent evaluation of four quantitative emergency department crowding scales. Acad Emerg Med 2006;13:1204–11.
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      1. Felton BM,
      2. Reisdorff EJ,
      3. Krone CN,
      4. et al
      . Emergency department overcrowding and inpatient boarding: a statewide glimpse in time. Acad Emerg Med 2011;18:1386–91.
      OpenUrlCrossRefPubMedWeb of Science

    Footnotes

    • Contributors AB and JC conceived the study. JC collected the data. AB, JC and PR analysed the data and wrote the report. YS, DK and JOK facilitated the study and supported data collection. All authors have read the final report.

    • Competing interests None.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    Linked Articles