Objective To determine the interrater reliability of triage acuity ratings by healthcare workers (HCW) using a previous triage system (PTS) and the Princess Marina Hospital accident and emergency centre triage scale (PATS), a local adaptation of the widely used and studied South African triage scale.

Methods A cross-sectional study was performed on HCW in an emergency department (ED) in Botswana to determine the interrater reliability of triage acuity ratings when using PTS and PATS to assign triage categories to 25 written vignettes after PATS training. The intraclass correlation coefficient (ICC) was calculated to assess interrater reliability, and graphic displays were used to portray rating distributions for vignettes with a mean rating of different acuity categories for PTS and PATS.

Results 44 HCW completed the scenarios. The ICC for the group of HCW was 0.52 (95% CI 0.37 to 0.67) using PTS and 0.87 (95% CI 0.80 to 0.93) using PATS. The ICC values were higher for PATS than PTS regardless of the number of years of experience of the HCW and the level of the HCW (specialist, medical officer, nurse, nurse aide). Graphic displays showed that there was less variability at all acuity levels when using PATS compared with PTS.

Conclusion The reliability measures in this study indicate very high interrater agreement and limited variability in acuity ratings when using the PATS as opposed to moderate agreement and increased variability in acuity ratings when using PTS. This suggests that PATS is reliably applied by all levels of HCW and supports the feasibility of the further implementation of PATS in ED in Botswana and in other similar settings.