Article Text
PDF
Abstract
Objective To determine the interrater reliability of triage acuity ratings by healthcare workers (HCW) using a previous triage system (PTS) and the Princess Marina Hospital accident and emergency centre triage scale (PATS), a local adaptation of the widely used and studied South African triage scale.
Methods A cross-sectional study was performed on HCW in an emergency department (ED) in Botswana to determine the interrater reliability of triage acuity ratings when using PTS and PATS to assign triage categories to 25 written vignettes after PATS training. The intraclass correlation coefficient (ICC) was calculated to assess interrater reliability, and graphic displays were used to portray rating distributions for vignettes with a mean rating of different acuity categories for PTS and PATS.
Results 44 HCW completed the scenarios. The ICC for the group of HCW was 0.52 (95% CI 0.37 to 0.67) using PTS and 0.87 (95% CI 0.80 to 0.93) using PATS. The ICC values were higher for PATS than PTS regardless of the number of years of experience of the HCW and the level of the HCW (specialist, medical officer, nurse, nurse aide). Graphic displays showed that there was less variability at all acuity levels when using PATS compared with PTS.
Conclusion The reliability measures in this study indicate very high interrater agreement and limited variability in acuity ratings when using the PATS as opposed to moderate agreement and increased variability in acuity ratings when using PTS. This suggests that PATS is reliably applied by all levels of HCW and supports the feasibility of the further implementation of PATS in ED in Botswana and in other similar settings.
- Emergency care systems
- emergency departments
- major incidents
- mass gathering medicine
- quality assurance
- remote and rural medicine
- systems
- triage
Statistics from Altmetric.com
- Emergency care systems
- emergency departments
- major incidents
- mass gathering medicine
- quality assurance
- remote and rural medicine
- systems
- triage
Despite the known benefits of effective triage, it is often performed poorly, especially in resource-limited settings. A standardised, effective triage system is critical in emergency medicine to prioritise patient care appropriately. Princess Marina Hospital (PMH) is the tertiary public hospital in Gaborone, Botswana. In 2009, the administrative leadership of the emergency department (ED) at PMH realised the need for an improved triage system. The previous triage system (PTS) was a three-level system based on a nurse's subjective assignment of each patient's condition as either life-threatening (I), potentially life-threatening (II), or non-life-threatening (III).
A multidisciplinary team of healthcare workers (HCW) at PMH, including the administrators, emergency medicine specialists, medical officers, nurses, nurse aides and an external consultant physician formed a triage implementation team. The implementation team reviewed validated triage systems and decided to adopt and adapt the South African triage scale (SATS),1 a four-level system that objectively categorises the medical urgency of ED patients based on age-appropriate physiological and clinical discriminators. Similar to the Manchester triage system2 and the Canadian emergency department triage and acuity scale,3 the SATS incorporates target times to treatment. Patients are categorised into one of four acuity levels including red (emergency—should be seen immediately), orange (very urgent—should be seen in under 10 min), yellow (urgent—should be seen in under 60 min), green (routine—should be seen in under 4 h) and blue (dead on arrival—should be certified within 2 h).4 The SATS has been implemented, monitored and modified in public emergency centres in South Africa since January 2006.5 It has been demonstrated in these settings to provide acceptable levels of undertriage and overtriage6 ,7 and was identified as an alternative to the PTS used at PMH in Botswana.
The SATS creators were recruited to provide SATS training for the implementation team in November 2009. The implementation team then made minor adaptations to the SATS using a modified Delphi process to optimise locally relevant clinical and linguistic factors. The new triage system was designated the PMH accident and emergency centre triage scale (PATS).
The primary objective of this study was to determine the interrater reliability of triage assignments for the PTS and PATS by all HCW on a set of written case vignettes. Secondary outcomes included determining the interrater reliability for different levels of HCW, different years of experience and different types of vignettes (adult and paediatric). The authors hypothesised that interrater reliability would be higher using a triage system based on objective criteria (PATS) rather than on subjective criteria (PTS).
Methods
Study design and sample
A cross-sectional study was conducted using a set of 25 vignettes that had been adapted from real patients (19 adults and six paediatric) and used in previous SATS studies.6 Vignettes were used as a suitable proxy for live triage cases.8 Based on their use in other studies9–16 and the advantage of saving cost and time,10 ,17 this method is particularly useful when resources are limited. Vignettes included gender, age, presenting complaint, mode of arrival, ambulatory status, mental status, respiratory rate, heart rate, temperature, blood pressure (for adult cases) and pertinent examination findings.
Sample size calculations were based on a SE of 0.05 and an intraclass coefficient (ICC) of 0.8 with at least 500 ratings. A convenience sample of HCW including nursing assistants, nurses, medical officers and emergency specialists, in the ED at PMH attended a 1-day training workshop on PATS. All HCW were asked independently to triage the 25 vignettes at the start of the workshop using PTS and at the end of the workshop using PATS. The guidelines for PTS were reviewed with staff before the vignette assessment with PTS.
Data analysis
Interrater reliability in this study was assessed with the ICC, which has been shown to be equivalent to the quadratically weighted κ (QWK).18 This summary has been used in other settings to measure agreement in different triage systems.6 Interrater reliability with 95% CI were assessed for each HCW level separately. Results were interpreted using the classification model designed by Landis and Koch.19 Graphic displays were constructed and have been used in other settings (M. Twomey, unpublished data 2009) to show rating distributions for vignettes with a mean rating of a particular acuity category.20 They have the ability to show differential agreement at different acuity categories, rather than summarising agreement in a single number. These analyses were performed using Stata (version 9.2) to calculate the ICC and illustrate reliability by graphic displays.
Results
A total of 44 HCW assessed 25 vignettes twice (1100 assessments using the PTS and 1100 assessments using the PATS). This convenience sample represented 96% (44 of 46) of the entire HCW team in the ED at PMH; two medical officers were on vacation. The ICC for all 44 HCW increased from a moderate agreement of 0.52 (95% CI 0.37 to 0.67) using PTS to a high agreement of 0.87 (95% CI 0.80 to 0.93) using PATS. Figure 1 illustrates that the reliability of ratings increased from moderate to high for adult and paediatric vignettes when using PATS as opposed to PTS. Table 1 summarises all other findings and shows that increases in reliability were consistent regardless of the years of experience and skills level of the HCW.
Summary of results on adult versus paediatric vignettes. PATS, Princess Marina Hospital accident and emergency centre triage scale; PTS, previous triage system.
Summary of secondary outcomes
Discussion
Excellent interrater reliability was observed for all HCW levels indicating that PATS provides reliable triage acuity ratings among adult and paediatric vignettes. This demonstrates that PATS is understood after a 1-day workshop and then consistently applied (in test vignettes) across all levels of HCW, regardless of years of experience or skills level. This finding suggests that any of the HCW who were trained in PATS could use this scale for emergency patient triage, and that task-shifting of triage should be considered and investigated depending on the relative availability of different cadres of HCW. The high reliability of PATS did not change depending on the number of years of professional experience of the HCW, suggesting that both new and well-experienced HCW could use the PATS effectively in triage situations.
A comparison between different triage scales is challenging as the number of categories within the ordinal scales varies.21 PATS (a four-level scale) and PTS (a three-level scale) are different tools and therefore not readily comparable, as the ICC, which is equivalent to the QWK,18 is dependent on the number of categories (ie, a higher number of categories in a scale increases the ICC).22 One expects to find a higher ICC for the ratings with PATS (four-level scale) than with PTS (three-level scale) and our study confirms this. The authors therefore decided to include an alternative exploratory graphic approach.
This illustration of interrater variability recommended by Nelson et al20 and used in other settings6 allows an assessment of agreement for different acuity levels and facilitates generalisability across study settings. It has been included here to allow a comprehensive assessment of reliability as well as to provide a more comparable exploratory visual assessment. When comparing the graphic displays of PTS in figures 2A–C and PATS in figures 3A–D, it is evident that less variability is found in rating distributions for different acuity levels when using PATS as opposed to PTS.
(A–C) Distribution of ratings for vignettes with mean ratings of life-threatening (n=6), potentially life-threatening (n=11) and non-life-threatening (n=8) when using the previous triage system (PTS). Illustrates the distribution of all healthcare worker (HCW) ratings when using the PTS. Vignettes with a mean rating of ‘life-threatening’ had 64% of ratings categorised as ‘life-threatening’ and the remainder of the ratings were below that; 62% of ratings were within the acuity level ‘potentially life-threatening’ for vignettes with a mean rating of ‘potentially life-threatening’ and the remaining ratings were equally distributed above and below. Vignettes with a mean rating of ‘non-life-threatening’ had 72% of ratings within that category.
(A–D) Distribution of ratings for vignettes with mean ratings of emergency (n=5), very urgent (n=8), urgent (n=8) and routine (n=4) when using the Princess Marina Hospital accident and emergency centre triage scale (PATS). Shows the rating distribution of healthcare worker (HCW) ratings using the PATS. For vignettes with a mean acuity category of ‘emergency’ and ‘very urgent’ 91% of ratings were within that category. Vignettes with a mean acuity level ‘urgent’ had 77% of ratings within the ‘urgent’ category and the remaining ratings were almost equally distributed one category above or below, and 89% of ratings were within the ‘routine’ acuity level for vignettes that had a mean acuity rating of ‘routine’.
Limitations
A limitation of any study assessing the reliability of triage scales using vignettes is that many non-verbal and contextual cues may be missed despite the fact that written vignettes are a very resource-efficient way of assessing reliability.16 ,17 We tried to overcome the lack of qualitative non-verbal cues by leaving the subjective information in the description of vignettes. Interrater reliability for PTS and PATS was assessed using ICC, which is equivalent to the QWK18 and has known limitations.21 ,22 The authors acknowledge that a direct comparison of different tools was not possible and that the ICC's dependence on the number of categories may have led to higher ICC in PATS. This was dealt with by including the graphic approach recommended by Nelson and Pepe20 In addition, there is a small possibility that the difference in interrater reliability between PTS and PATS are due solely to a training effect. Addressing this limitation in the study design would not have been practical, as it may have entailed conducting a full day workshop on the PTS when it was already known that it was flawed and needed to be replaced. Furthermore, there was a possibility that using the same 25 vignettes to assess HCW before and after training may have led to a familiarity effect, which in turn favoured PATS.
Conclusion
This is the first interrater reliability study of triage systems derived from SATS, and highlights that the PATS presents reliable triage acuity ratings and has less variability than PTS in rating distributions for different acuity levels. The reliability of PATS was high regardless of the years of experience of the HCW, the level of HCW, and whether adult or paediatric vignettes were used. This study supports the feasibility of the potentially wider implementation of PATS in other hospital settings in Botswana and beyond.
Acknowledgments
The authors would like to acknowledge the involvement of Beauty Sepelete, Kolobetso Malatsi, Ibrahim Sani, Jephres Muzondo and Lefika Kareng who helped make this project possible as the implementation team at Princess Marina Hospital.
References
Footnotes
Competing interests None.
Provenance and peer review Not commissioned; internally peer reviewed.