ResearchTriage Tool Inter-rater Reliability: A Comparison of Live Versus Paper Case Scenarios
Section snippets
Objective
This study was designed to assess the correlation between triage assessments conducted on patients in a live ED setting and those assigned to equivalent paper case scenarios. In addition, we aimed to compare the inter-rater reliability of the tool across both presentation media.
Ethics
This study protocol was submitted to the local Ethics Review Board and found to be exempt from formal ethics review because it was considered a quality assurance study that did not interfere with or alter standard
Results
Triage data were collected (and turned into paper-based cases) for a total of 271 patients. The IRR for the average rating assigned by 3 triage nurses when viewing the cases live was 0.9 (95% CI = 0.89-0.91). When the same cases were reviewed in paper format, the IRR for the average rating assigned by 3 triage nurses was 0.76 (95% CI = 0.74-0.78). The correlation between the research nurses' live triage assessments and the triage assessments they assigned to the paper cases was r = 0.73 (95% CI
Discussion
This study was designed to ensure that selected patients were representative of actual ED case mixes and employed “experienced” triage nurses representative of those performing triage assessments in other hospitals.
The advantages of triage researchers using paper case scenarios are in the savings of cost and time; rather than have a few triage nurses assess a few patients over a long period, multiple cases can be assessed by a large group of observers in a few hours. Although the results show a
Limitations
A limitation of this study is that the research triage nurses were not permitted to directly question patients, because this questioning would interfere with standard patient care. This restriction may have affected their initial triage assessment. However, the level of agreement found between the research nurses' triage assessments and actual triage nurse's assessments was high at 0.71. Also, the scoring of paper case scenarios similarly precluded acquiring additional information other than
Conclusions
Use of paper case scenarios to determine IRR of triage scales is an efficient method that estimates that of live cases. If the results are found to be within an acceptable performance range, further testing of IRR using live cases may be unnecessary.
Andrew Worster is Research Director, Division of Emergency Medicine, McMaster University and Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.
References (9)
- et al.
Paper case scenarios in the assessment of triage tools
Ann Emerg Med
(2005) - et al.
Reliability of the Canadian Emergency Department Triage and Acuity Scale: inter-observer agreement
Ann Emerg Med
(1999) Emergency department triage: why we need a research agenda
Ann Emerg Med
(2004)- et al.
Health Measurement Scales: a practical guide to their development and use (3rd ed.)
(2003)
Cited by (55)
A descriptive study of registered nurses’ application of the triage scale RETTS©; a Swedish reliability study
2018, International Emergency NursingCitation Excerpt :On the other hand, using paper scenarios guarantees that the RNs had access to the same information. Others have argued that paper scenarios represent an efficient method to study inter-rater reliability compared to real-life patients; it saves time and costs [36], and this limitation is therefore open to discussion. The majority (85%) of the 46 scenarios were triaged with dispersion.
Validity of ED: Addressing heterogeneous definitions of over-triage and under-triage
2017, American Journal of Emergency MedicineAcuity Assessment in Obstetrical Triage
2016, Journal of Obstetrics and Gynaecology CanadaPredicting Pediatric Emergency Severity Index Level Based on Emergency Department Pre-Arrival Information
2018, Journal of Pediatric NursingFirst responder accuracy using SALT after brief initial training
2015, Prehospital and Disaster Medicine
Andrew Worster is Research Director, Division of Emergency Medicine, McMaster University and Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.
Arlene Sardo is Nurse Practitioner, Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.
Kevin Eva is Epidemiologist, Department of Clinical Epidemiology and Biostatics, McMaster University, Hamilton, Ontario, Canada.
Christopher M.B. Fernandes is Professor of Emergency Medicine, University of Western Ontario, London, Ontario.
Suneel Upadhye is Undergraduate Coordinator, Division of Emergency Medicine, McMaster University and Hamilton Health Sciences, Department of Emergency Medicine, Hamilton, Ontario, Canada.
This study was presented at the 2005 Society of Academic Emergency Medicine Annual Meeting, May 22-25, 2005, New York, NY; the Canadian Association of Emergency Physicians Annual Scientific Meeting, May 30, 2005, Edmonton, Alberta; and the ENA Annual Conference, Nashville, Tenn, September 14-17, 2005.
Funded by a research grant to Dr. Andrew Worster from the Hamilton Emergency Services Network, Hamilton Ontario, Canada.
Authors' note
This study has been published in abstract form as follows:
Sardo A, Worster A, Fernandes CMB, Eva K, Upadhye S. Triage tool inter-rater reliability using live cases vs. paper case scenarios. Acad Emerg Med 2005;12:137.
Worster A, Sardo A, Fernandes CMB, Eva K, Upadhye S. Triage tool inter-rater reliability using live cases vs. paper case scenarios. Can J Emerg Med 2005;7:209.
Worster A, Sardo A, Fernandes C, Eva K, Upadhye S. Triage tool inter-rater reliability using live cases vs. paper case scenarios. J Emerg Nurs 2005;31:426.