Table 4

Reference standards developed by investigators for validating triage systems

AuthorYearTriage systemsDesignValidated sampleDetails of reference standards
Dong2005CTASProspectivePatientsAn expert panel determined triage levels of randomly selected patients through consensus after chart reviews, while they were kept blinded from the bedside assessment, investigation, management and patient outcomes.
Roukema2006MTSRetrospectivePatientsAn expert panel defined a five-level reference classification for true urgency. The classification comprised a combination of vital signs, a possible life-threatening condition, resource use (diagnostic investigation and treatment) and disposition from the ED.
van Veen2008MTSProspectivePatientsPaediatricians and a paediatric surgeon defined a five-level reference standard based on the literature and expert opinions. This was based on a combination of vital signs, diagnosis, diagnostic and therapeutic interventions, hospital admission and follow-up.
van Veen2012MTS and modified MTSProspectivePatientsPaediatricians and a paediatric surgeon created a reference urgency category based on a combination of vital signs, diagnosis, diagnostic and therapeutic interventions and hospitalisation/follow-up.
Storm-Versloot2014MTS, ESI, ISSProspectivePatientsAn expert panel comprising seven experienced ED physicians set a five-level urgency classification. Each physician independently reviewed each case record and determined a category based on ED data, results of diagnostic tests, final diagnosis and patient prognosis.
Steiner2016MTSProspectivePatientsTwo independent physicians determined an urgency level based on the clinical data, diagnostic tests and final diagnosis.
van der Wulp2008MTSProspectiveVignettesTwo experts independently evaluated vignettes of 50 actual patients and assigned degrees of urgency through consensus.
Olofsson2009MTSProspectiveVignettesAn expert panel rated and determined degrees of urgency in case scenarios through consensus.
Ruipérez2015ESIProspectivePatientsTriage nurses and a triage expert independently evaluated patients, and the expert’s evaluation was considered as the reference standard.46
  • CTAS, Canadian Triage and Acuity Scale; ESI, Emergency Severity Index; ISS, informally structured triage system; MTS, Manchester Triage System.