You are here

Article Text

Article menu
Original article
The Manchester triage system: improvements for paediatric emergency care
  1. Mirjam van Veen1,
  2. Ewout W Steyerberg2,
  3. Mariët van't Klooster1,
  4. Madelon Ruige3,
  5. Alfred H J van Meurs3,
  6. Johan van der Lei4,
  7. Henriëtte A Moll1
  1. 1Department of Paediatrics, Erasmus MC-Sophia Children's Hospital, Rotterdam, The Netherlands
  2. 2Center for Medical Decision Making, Public Health, Erasmus MC, Rotterdam, The Netherlands
  3. 3Department of Paediatrics, Haga Hospital, Juliana Children's Hospital, The Hague, The Netherlands
  4. 4Department of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands
  1. Correspondence to Professor Henriëtte Moll, Department of Paediatrics, Room Sp 1540, Erasmus MC-Sophia Children's Hospital, PO Box 2060, Rotterdam 3000 CB, The Netherlands; h.a.moll{at}erasmusmc.nl

Abstract

Objective To improve the Manchester Triage System (MTS) in paediatric emergency care.

Methods The authors performed a prospective observational study at the emergency departments of a university and teaching hospital in The Netherlands and included children attending in 2007 and 2008. The authors developed and implemented specific age-dependent modifications for the MTS, based on patient groups where the system's performance was low. Nurses applied the modified system in 11 481 (84%) patients. The reference standard for urgency defined five levels based on a combination of vital signs at presentation, potentially life-threatening conditions, diagnostic resources, therapeutic interventions and follow-up. The reference standard for urgency was previously defined and available in 11 260/11 481 (96%) patients.

Results Compared with the original MTS specificity improved from 79% (95% CI 79% to 80%) to 87% (95% CI 86% to 87%) while sensitivity remained similar ((63%, 95% CI 59% to 66%) vs (64%, 95% CI 60% to 68%)). The diagnostic OR increased (4.1 vs 11).

Conclusions Modifications of the MTS for paediatric emergency care resulted in an improved specificity while sensitivity remained unchanged. Further research should focus on the improvement of sensitivity.

  • Triage
  • epidemiology
  • emergency medical services
  • child
  • emergency department
  • paediatrics
  • paediatric emergency medicine

https://doi.org/10.1136/emermed-2011-200562

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Triage is an important tool to manage patient flow safely when clinical need exceeds capacity.1 Several triage systems have been developed and applied in emergency care. The Manchester Triage System (MTS), the Emergency severity Index, the Canadian Triage and acuity scale and the Australasian Triage scale are commonly used triage systems.1–6 They all use specific criteria to triage children.

    The MTS consists of 52 flowcharts, which present reasons of encounter, of which 49 are applicable for children. Each flowchart contains discriminators of which selection leads to one out of the five urgency levels. An example of a MTS flowchart is provided in figure 1. In adults, the MTS was shown to be sensitive for early detection of seriously ill patients and for the detection of high-risk chest pain.7–9 The MTS is not valid to detect children with serious bacterial infections.10 The MTS showed a substantial to good inter-rater agreement in adults and children.11 ,12

    We studied the validity of the MTS in children and showed a moderate validity with a sensitivity of 63% (95% CI 59% to 66%) and a specificity of 79% (79% to 80%) for identifying high urgency patients. Performance was especially low for patients with a non-traumatic presenting problem, for patients with a young age and for patients with fever.13

    The aim of this study was to improve the MTS for paediatric emergency care based on modifications in patient groups in which the MTS performance was low and to evaluate its performance in a new population.

    Methods

    Study design

    We performed an observational prospective study. Modifications were developed for patient subgroups where the MTS performance was low.13 The modified MTS was implemented at two emergency departments (EDs). We evaluated performance by comparing the MTS urgency levels with a predefined reference standard for urgency. Improvements were evaluated by comparing performance of the modified MTS with performance of the original MTS, as evaluated in our previous study.13

    Setting and selection of participants

    The Erasmus MC-Sophia Children's hospital in Rotterdam, The Netherlands, is a university hospital with a specific paediatric ED that receives 9000 patient-visits per year. The Haga hospital- Juliana children's Hospital at The Hague, a large urban teaching hospital with a full spectrum of patients, encounters 15 000 paediatric patient-visits per year. Patients aged <16 years were included in the two hospitals, between May 2007–April 2008 and August–December 2007, respectively.

    Methods of measurement: MTS

    ED nurses performed a short assessment and triaged patients using the MTS. First, the patient's presenting problem will be identified; it leads to the selection of a flowchart. An example of an MTS flowchart is provided in figure 1. MTS flowcharts contain six general discriminators (life threat, pain, haemorrhage, acuteness of onset, level of consciousness and temperature) as well as specific discriminators, which are relevant to the presenting problem. Selection of a discriminator leads to the same urgency level independent of the selected flowchart. The selected discriminator indicates one of the five urgency categories, with a maximum waiting time (‘immediate’ 0 min, ‘very urgent’ 10 min, ‘urgent’ 60 min, ‘standard’ 120 min and ‘non-urgent’ 240 min). Pain is scored on a scale from 0 to 10 and can assign patients to a higher urgency level. Nurses used a digital version of the MTS to triage patients.

    Figure 1
    Figure 1

    Manchester Triage System flowchart: shortness of breath in children.

    The triage nurses were all specialised ED, paediatric nurses. They followed the Manchester Triage Course and the Emergency Nursing Paediatric Course of the Emergency Nursing Association.

    Outcome measures

    Prior to the study, a reference standard was defined based on literature and expert opinion.13 A reference urgency was defined based on a combination of vital signs, diagnosis, diagnostic and therapeutic interventions and hospitalisation/follow-up. It was developed prior to the onset of the study in an expert meeting by paediatricians and a paediatric surgeon.

    Patients were considered to be category 1 if their vital signs were deviated according to the limits used in the Pediatric Risk of Mortality Score14 or in case of hyperthermia (temperature >41°C). Patients with hyperthermia have a higher risk for a severe bacterial infection.15 Deviations in temperature, respiratory rate or pulse oximetry and mental status are related to resource use and hospitalisation.16 ,17

    Nurses fully examined all children; vital signs were measured at the discretion of the nurse or physician and were not measured if children appeared not ill or had no signs of respiratory or circulatory distress. For those presenting with a medical (non-trauma) problem, temperature was measured in 81%, heart rate in 49% and respiratory rate in 28% of cases. If vital signs were not measured, they were assumed to be normal.

    Patients were assigned to category 2 if they had normal vital signs and their presumed diagnosis at the end of their ED consultation was defined as a potential life-threatening condition.13 The potential life-threatening conditions are provided in the online appendix 1. Most of these conditions are associated with a high morbidity and mortality and are discussed in the Advanced Paediatric Life Support workbook as an emergency.18 ,19 Patients presenting with an apparently life-threatening event were suggested to be monitored for 24 h; we therefore considered them as very urgent.20

    Patients were allocated to category 3 or 4 depending on the performed diagnostics, administered therapy, hospitalisation and if a follow-up visit was scheduled.

    Category 5 was defined if patients did not require any of the resources. Previous studies on other triage systems for paediatric patients showed an association between urgency level and resource use and follow-up. Resource use is associated with the urgency level of the Emergency Severity Index.21–23

    A classification matrix of the reference classification and detailed definitions of the reference standard urgencies were published before and are provided in online appendix 1.13 We defined the reference standard for each patient independent of the MTS urgency and based on a computerised application of the classification matrix. We defined overtriage and undertriage when the MTS urgency level was higher and lower, respectively, than the reference standard urgency level.

    Modifications

    We studied patient characteristics and their relation to agreement between the MTS and the reference standard urgency distribution.13 Second, modifications were discussed with the ED nurses and paediatricians of both hospitals. The aim of modifying the MTS was to increase correct triage and decrease overtriage without increasing undertriage.

    Modifications focused on the patient characteristics: a) age, referral status and presenting problem (traumatic or medical), b) frequently used MTS flowcharts, c) frequently used MTS discriminators and d) combination of a–c. The proportion of number of categories of overtriage and undertriage was calculated for subgroups based on these patient characteristics to evaluate whether attributing a higher or a lower MTS urgency category could result in a better agreement between the MTS and the reference standard urgency. Modifications were created for subgroups if in ≥80% of the patients the reference urgency category was lower than ‘very urgent’.

    The MTS had a low performance for patients with fever, in particular.13 As an example, the rationale for the modification for patients with fever is shown in figure 2.

    Figure 2
    Figure 2

    Reference standard urgency distribution for Manchester Triage System flowcharts and age subgroups for patients triaged with a fever discriminator. Modifications on MTS flowchart and age subgroups (m=months, y=years) were developed for patients with fever. All patients were triaged into the ‘Very urgent’ (gray) category according to the original MTS. Modifications were created for subgroups if in >=80% of the patients the reference urgency category agreed with a lower urgency category. (No patients were included aged < 3 months with the MTS flowchart ‘Rashes’ or ‘Sore throat’.) Percentages above 4% are shown in the figure.

    One new discriminator was inserted after discussion of the modified MTS among the paediatricians working at the participating EDs. For the flowchart ‘Rashes’, a new discriminator ‘Petechiae’ was inserted in the ‘very urgent’ (level 2) category, in order to detect patients with petechiae who are highly suspect of meningococcal septiceamia.

    Other modifications were developed using the same method. However, other cut-off levels, as discussed by experts, were used for allocating patients in lower urgency categories. An overview of the modifications with cut-off levels is provided in table 1.

    View this table:
    Table 1

    Modifications for the Manchester triage system (MTS) in paediatric emergency care

    External validation

    The modified MTS was implemented at the ED of both hospitals. Before implementation, meetings were conducted to instruct the ED nurses on the details of the modifications. A detailed description of the modification was provided. We performed monthly audits on the compliance of the use of the MTS which remained appropriate (76%–96%). Feedback of the compliance was given to the director of the EDs.

    We studied reliability by assessing the inter-rater agreement. Twenty-one ED nurses of both studied hospitals classified the urgency level of the same 30 written case scenarios using the modified MTS. The written case scenarios were obtained and selected from other studies on reliability of triage systems for children24 ,25 and six local scenarios were added. We aimed to evenly distribute age, presenting problems and acuity level (according to the Emergency Severity Index). The selected urgency level per case scenario was compared between the nurses. We used the intraclass correlation coefficient as a measure of inter-rater agreement, which can be used for multiple raters and can be interpreted as quadratic weighted κ, as was used before to study reliability of the original MTS in children.12 The inter-rater agreement in acuity level (quadratic weighted κ) 0.77 (95% CI 0.67 to 0.86) was similar to the reliability of the original MTS (0.83, 95% CI 0.74 to 0.91).12

    Data collection

    Patient characteristics, selected MTS flowchart, MTS discriminator and the urgency category were recorded in the computerised triage system. Urgency levels according to the original MTS could be determined by the selected MTS flowchart and discriminator. Nurses or physicians recorded data concerning vital signs, diagnosis, diagnostic and therapeutic interventions, hospitalisation and follow-up on structured electronic ED forms. Trained medical students gathered and entered the data for the reference standard, independent of the triage outcome, using SPSS-Data Entry V.4. The database was checked for consistency and outliers. Data on laboratory tests were obtained from the hospital information system.

    Primary data analysis

    To assess performance of the modified MTS we calculated percentage over-, under and correct triage. Second, we calculated sensitivity, specificity and the diagnostic OR (DOR). Patients were categorised as high urgent (level 1 or 2) or low urgent (level 3, 4 or 5). The DOR describes the odds of an MTS high urgency in high urgency patients, according to the reference standard, as compared with the odds of an MTS high urgency in low urgency patients, according to the reference standard (DOR=(sensitivity/1-sensitivity)/(1-specificity/specificity)).26

    A risk stratification table shows the extent to which high urgency patients, according to the reference standard, are assigned to MTS high urgency categories. Second, it shows if low urgency patients, according to the reference standard, are assigned to MTS low urgency categories27 (table 2). SPSS V.15.0 was used for statistical analysis.

    View this table:
    Table 2

    Original Manchester Triage System (MTS) compared with modified MTS and proportion of high urgency patients as classified by the reference standard for urgency in dataset 2 (2007–2008)

    Results

    Characteristics of study subjects

    In total, 13 654 patients presented at the ED of the Sophia Children's Hospital and Juliana Children's Hospital during the study period. In all, 11 927 (84%) were triaged using the modified MTS. In the Sophia Children's hospital, the nurses had the possibility to over-rule the system, and did so in 3.4% (221/6531) of the patients; these patients were excluded. The reference standard for urgency could be determined in 11 260 patients (96%).

    The median age was 3.4 years (IQR 1.2–8.2), 4748 (42%) were female subjects and 5112 (45%) were referred by a healthcare professional. Patients were triaged using the flowcharts ‘Shortness of breath in children’ in 13.9%, ‘General’ in 11.2%, ‘Vomiting or diarrhoea’ in 8.7%, ‘Worried parent’ in 7.8%, ‘Abdominal pain in children’ in 5.9%, ‘Rashes’ in 2.7%, ‘Fits’ in 2.7%, ‘Urinary problems’ in 2.4%, ‘Ear problems’ in 1.7% and 13.7% of the patients were triaged with other medical flowcharts. Flowcharts for traumatic problems were applied in 29.3%. These characteristics were comparable with patients included in the first dataset.13

    Main results

    The urgency levels of patients triaged using the modified MTS were compared with the five reference standard urgency levels. The modified MTS agreed in 37% (n=4204) with the reference standard urgency, 36% (n=4091) were overtriaged by one category and 11% (n=1276) by more than one category. About 13% (n=1477) were undertriaged by one category and 2% (n=212) by >1 category.

    Sensitivity was 64% (95% CI 60% to 68%) and specificity 87% (95% CI 86% to 87%), resulting in a DOR of 11.5 (95% CI 9.6 to 14) (table 3).

    View this table:
    Table 3

    Performance of original and modified Manchester Triage System (MTS)

    Due to the modifications, 930 patients (8.2%) were reclassified to other urgency categories. For the three highest original MTS urgency categories, all reclassified patients were allocated to a lower urgency category. The reclassified patients who were allocated to a lower urgency category had a lower incidence of high urgency cases according to the reference standard, compared with patients who were not reclassified (2.2% vs 5.7%). Table 2 provides the reclassification of patients from the original MTS urgency to the modified MTS urgency per urgency level.

    Discussion

    This study shows that some small modifications resulted in an improved version of the MTS in paediatric emergency care. Specificity improved and sensitivity remained similar. The DOR improved. The modifications reclassified 8.2% of the patients.

    Misclassification may partly be explained by incorrect application of the system. Therefore, we checked the reliability of the modified MTS. The inter-rater agreement was similar when compared with the inter-rater agreement of the original MTS (Kw 0.77, 95% CI 0.67 to 0.86) indicating that the modified MTS was correctly applied in most cases.

    Since triage systems are usually developed by expert opinion, it is important to evaluate its performance and modify the system if performance is not optimal.

    We identified patient groups where the MTS has a low performance for children and developed modifications based on and for these patient groups. The modifications were implemented and evaluated in a new population in two ED settings. As was shown for clinical prediction models, even if a model is derived from patient data (instead of expert opinion), it is important to validate it in a new population since generalisability of models with good internal validation measures can be very disappointing. This can be caused by inadequate development of the model or major differences between the populations in which the model was developed and validated.28 Our validation set was comparable with the first dataset (2006–2007) on age, gender, flowchart distribution and reference standard urgency classification. Therefore, we can assume that improvement in performance is mostly due to the modification and less to a change in population characteristics.

    The Manchester triage group developed modifications in 2006 to improve the system for children and adults.4 These modifications were developed by comments from users and were not validated.

    Specific modifications for children were developed in other common used triage systems in paediatric emergency care as well. To our knowledge, only one study evaluated the effects of one of these modifications. For the paediatric version of the level 5 Canadian Triage and Acuity Scale,3 children with fever were assigned to a lower urgency category (urgency category 4 instead of urgency category 3) if they had no signs of toxicity (defined as unexplained crying before examination, difficulty awakening or poor response to the physical evaluation) and were older than 6 months of age. Admission rates were compared between patients who remained in level 3 and the modified patients. The modified patients had lower admission rates and none died or required Intensive Care Unit admission.29

    Other studies evaluated performance of triage systems in children and employed resource use, hospitalisation, length of stay or Intensive Care Unit admission as outcome.21 ,22 ,30 ,31 However, these studies did not aim to identify specific patient groups in which the studied triage system performed less. Two studies evaluated performance of triage systems in adults. They identified patient groups based on age, sex, presenting problem and vital signs, in which performance was low, in order to develop modifications that could improve performance of triage systems.32 ,33

    After modification, the performance of the MTS improved. The modified MTS showed a specificity of 87% and a sensitivity of 64%. Users should realise that although the high specificity indicates that the system has a good ability to identify low urgent patients, the sensitivity is only moderate.

    Although we improved the systems, further improvement in performance, especially sensitivity, should be developed. Since the MTS uses descriptions of vitally threatened patients instead of concrete limits for vital signs to be concerned as deviated, some high urgency patients will be missed. For example, patients with ‘shock’ are defined as level 1 (‘immediate’), patients with a tachycardia, arrhythmia or deviated blood pressure without shock are not considered as high urgent, unless other high urgent discriminators are present.4 Our previous study showed that severe undertriage was rare using the MTS; however, 89% of undertriaged children had abnormal vital signs, which the MTS had not recognised.34

    Large datasets are necessary to identify patient characteristics and discriminators which predict urgent with a high discriminative value.

    Limitations

    In the studied population, 45% were referred by a general practitioner. This proportion is comparable with referral rates in other settings where primary care physicians are available, with referral rates ranging from 30% to 80% (European countries,35–38 Australia,39 ,40 the USA41 ,42 and Canada43 ,44). Therefore, we think that are results are generalisable to these other populations as well.

    We developed a proxy for urgency which was determined out of characteristics of the patients' ED consultation (vital signs, life-threatening conditions, performed diagnostics and therapy, scheduled follow-up or hospitalisation). The individual items predict severity of disease and were combined by expert opinion to define a reference standard for urgency (five levels). Although it is not a real ‘gold standard’, it is a proxy for ‘real urgency’.13 ,45 No other outcome measure for urgency that defines five urgency levels is previously described in the literature.

    Modifications were developed after applying the original MTS in 17 600 patients in 2006–2007 who visited the ED of a large teaching hospital or a university children's hospital in The Netherlands. We aimed to develop modifications resulting in an optimal performance of the MTS and to minimise undertriage and overtriage. Therefore, we had to develop modified discriminators which allocate a patient to different urgency levels if different flowcharts are used. (For example, a child aged between 2 months and 3 years with fever will have a ‘Urgent’ triage level when triaged using the flowchart ‘Worried parent’, ‘General’ or ‘Urinary problems’ while a ‘Standard’ triage level is attributed when the child is triaged using the flowchart ‘Sore throat’, ‘Ear problems’ or ‘Rashes’.) If a computerised system is used, these modifications can easily be implemented.

    Evaluation of performance of the modifications was done in a new population. The modified MTS was applied in 13 654 patients in 2007–2008. Therefore, our results are representative for a wide population of children presenting at the ED. However, since the modified MTS was applied in the same setting in which the modifications were developed, the modifications should be validated in different settings as well.

    Conclusion

    Modifications of the MTS for paediatric emergency care resulted in an improved specificity while sensitivity remained similar. The DOR also improved substantially. The modified MTS should be validated in different settings to further evaluate its performance. In all, 8.2% of the cases were reclassified according to the modified MTS; therefore, extended data collections are necessary to further improve the MTS in paediatric emergency care.

    Acknowledgments

    We thank the ED nurses of the Erasmus MC-Sophia children's hospital and the Haga Hospital-Juliana Children's hospital for their cooperation in this study, Marcel de Wilde, BSc, Department of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands, for technical support. Furthermore, we thank Csila de Knijff, medical student, for her participation in the data collection.

    References

    1. ↵
      1. Mackway-Jones K
      . Emergency Triage, Manchester Triage Group. London: BMJ Publishing Group, 1997.
    2. ↵
      Australian College for Emergency Medicine. Guidelines on the Implementation of the Australasian Triage Scale in Emergency Departments. http://www.acem.org.au/media/policies_and_guidelines/G24_Implementation__ATS.pdf
    3. ↵
      1. Beveridge R
      . CAEP issues. The Canadian Triage and Acuity Scale: a new and critical element in health care reform. Canadian Association of Emergency Physicians. J Emerg Med 1998;16:507–11.
      OpenUrlCrossRefPubMed
    4. ↵
      1. Mackway-Jones K,
      2. Marsden J,
      3. Windle J
      , eds. Emergency Triage, Manchester Triage Group. 2nd edn. Oxford: Blackwell Publishing Ltd, 2006.
    5. ↵
      1. Walls R
      . Dr Richard Wuerz's emergency severity index. Acad Emerg Med 2001;8:183–4.
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Wuerz RC,
      2. Travers D,
      3. Gilboy N,
      4. et al
      . Implementation and refinement of the emergency severity index. Acad Emerg Med 2001;8:170–6.
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Cooke MW,
      2. Jinks S
      . Does the Manchester triage system detect the critically ill? J Accid Emerg Med 1999;16:179–81.
      OpenUrlAbstract/FREE Full Text
    8. ↵
      1. Pinto D,
      2. Lunet N,
      3. Azevedo A
      . Sensitivity and specificity of the Manchester Triage System for patients with acute coronary syndrome. Rev Port Cardiol 2010;29:961–87.
      OpenUrlPubMed
    9. ↵
      1. Speake D,
      2. Teece S,
      3. Mackway-Jones K
      . Detecting high-risk patients with chest pain. Emerg Nurse, 2003;11:19–21.
      OpenUrlPubMed
    10. ↵
      1. Nijman RG,
      2. Zwinkels RL,
      3. van Veen M,
      4. et al
      . Can urgency classification of the Manchester triage system predict serious bacterial infections in febrile children? Arch Dis Child 2011;96:715–22.
      OpenUrlAbstract/FREE Full Text
    11. ↵
      1. van der Wulp I,
      2. van Baar ME,
      3. Schrijvers AJ
      . Reliability and validity of the Manchester Triage System in a general emergency department patient population in the Netherlands: results of a simulation study. Emerg Med J 2008;25:431–4.
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. van Veen M,
      2. Teunen-van der Walle VF,
      3. Steyerberg EW,
      4. et al
      . Repeatability of the Manchester Triage System for children. Emerg Med J 2010;27:512–16.
      OpenUrlAbstract/FREE Full Text
    13. ↵
      1. van Veen M,
      2. Steyerberg EW,
      3. Ruige M,
      4. et al
      . Manchester triage system in paediatric emergency care: prospective observational study. BMJ 2008;337:a1501.
      OpenUrlAbstract/FREE Full Text
    14. ↵
      1. Pollack MM,
      2. Patel KM,
      3. Ruttimann UE
      . PRISM III: an updated pediatric risk of mortality score. Crit Care Med 1996;24:743–52.
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Trautner BW,
      2. Caviness AC,
      3. Gerlacher GR,
      4. et al
      . Prospective evaluation of the risk of serious bacterial infection in children who present to the emergency department with hyperpyrexia (temperature of 106 degrees F or higher). Pediatrics 2006;118:34–40.
      OpenUrlAbstract/FREE Full Text
    16. ↵
      1. Chamberlain JM,
      2. Patel KM,
      3. Pollack MM
      . The pediatric risk of hospital admission score: a second-generation severity-of-illness score for pediatric emergency patients. Pediatrics 2005;115:388–95.
      OpenUrlAbstract/FREE Full Text
    17. ↵
      1. Gorelick MH,
      2. Lee C,
      3. Cronan K,
      4. et al
      . Pediatric emergency assessment tool (PEAT): a risk-adjustment measure for pediatric emergency patients. Acad Emerg Med 2001;8:156–62.
      OpenUrlPubMedWeb of Science
    18. ↵
      1. Mackway-Jones K,
      2. Molyneux E,
      3. Phillips B,
      4. et al
      ; Advanced Life Support Group. Advanced Paediatric Life Support. 3rd edn. London; BMJ Books, 2001.
    19. ↵
      1. Armon K,
      2. Stephenson T,
      3. MacFaul R,
      4. et al
      . An evidence and consensus based guideline for acute diarrhoea management. Arch Dis Child 2001;85:132–42.
      OpenUrlAbstract/FREE Full Text
    20. ↵
      1. McGovern MC,
      2. Smith MB
      . Causes of apparent life threatening events in infants: a systematic review. Arch Dis Child 2004;89:1043–8.
      OpenUrlAbstract/FREE Full Text
    21. ↵
      1. Baumann MR,
      2. Strout TD
      . Evaluation of the Emergency Severity Index (version 3) triage algorithm in pediatric patients. Acad Emerg Med 2005;12:219–24.
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Gouin S,
      2. Gravel J,
      3. Amre DK,
      4. et al
      . Evaluation of the Paediatric Canadian Triage and Acuity Scale in a pediatric ED. Am J Emerg Med 2005;23:243–7.
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. Maningas PA,
      2. Hime DA,
      3. Parker DE
      . The use of the Soterion Rapid Triage System in children presenting to the Emergency Department. J Emerg Med 2006;31:353–9.
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Storm-Versloot MN,
      2. Ubbink DT,
      3. Chin a Choi V,
      4. et al
      . Observer agreement of the Manchester Triage System and the Emergency Severity Index: a simulation study. Emerg Med J 2009;26:556–60.
      OpenUrlAbstract/FREE Full Text
    25. ↵
      1. Worster A,
      2. Gilboy N,
      3. Fernandes CM,
      4. et al
      . Assessment of inter-observer reliability of two five-level triage and acuity scales: a randomized controlled trial. CJEM 2004;6:240–5.
      OpenUrlPubMed
    26. ↵
      1. Glas AS,
      2. Lijmer JG,
      3. Prins MH,
      4. et al
      . The diagnostic odds ratio: a single indicator of test performance. J Clin Epidemiol 2003;56:1129–35.
      OpenUrlCrossRefPubMedWeb of Science
    27. ↵
      1. Janes H,
      2. Pepe MS,
      3. Gu W
      . Assessing the value of risk predictions by using risk stratification tables. Ann Intern Med 2008;149:751–60.
      OpenUrlCrossRefPubMedWeb of Science
    28. ↵
      1. Toll DB,
      2. Janssen KJ,
      3. Vergouwe Y,
      4. et al
      . Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol 2008;61:1085–94.
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      1. Gravel J,
      2. Manzano S,
      3. Arsenault M
      . Safety of a modification of the triage level for febrile children 6 to 36 months old using the Paediatric Canadian Triage and Acuity Scale. CJEM 2008;10:32–7.
      OpenUrlPubMed
    30. ↵
      1. Gravel J,
      2. Manzano S,
      3. Arsenault M
      . Validity of the Canadian Paediatric Triage and Acuity Scale in a tertiary care hospital. CJEM 2009;11:23–8.
      OpenUrlCrossRefPubMed
    31. ↵
      1. Ma W,
      2. Gafni A,
      3. Goldman RD
      . Correlation of the Canadian Pediatric Emergency Triage and Acuity Scale to ED resource utilization. Am J Emerg Med 2008;26:893–7.
      OpenUrlCrossRefPubMed
    32. ↵
      1. Ruger JP,
      2. Lewis LM,
      3. Richter CJ
      . Identifying high-risk patients for triage and resource allocation in the ED. Am J Emerg Med 2007;25:794–8.
      OpenUrlCrossRefPubMedWeb of Science
    33. ↵
      1. Tanabe P,
      2. Travers D,
      3. Gilboy N,
      4. et al
      . Refining Emergency Severity Index triage criteria. Acad Emerg Med 2005;12:497–501.
      OpenUrlCrossRefPubMedWeb of Science
    34. ↵
      1. Seiger N,
      2. van Veen M,
      3. Steyerberg EW,
      4. et al
      . Undertriage in the Manchester triage system: an assessment of severity and options for improvement. Arch Dis Child 2011;96:653–7.
      OpenUrlAbstract/FREE Full Text
    35. ↵
      1. Huibers L,
      2. Giesen P,
      3. Wensing M,
      4. et al
      . Out-of-hours care in western countries: assessment of different organizational models. BMC Health Serv Res 2009;9:105.
      OpenUrlCrossRefPubMed
    36. ↵
      1. Laffoy M,
      2. O'Herlihy B,
      3. Keye G
      . A profile of attenders to a south Dublin city accident and emergency department. Ir J Med Sci 1997;166:35–7.
      OpenUrlCrossRefPubMed
    37. ↵
      1. Philips H,
      2. Remmen R,
      3. De Paepe P,
      4. et al
      . Out of hours care: a profile analysis of patients attending the emergency department and the general practitioner on call. BMC Fam Pract 2010;11:88.
      OpenUrlPubMed
    38. ↵
      1. Thompson M,
      2. Coad N,
      3. Harnden A,
      4. et al
      . How well do vital signs identify children with serious infections in paediatric emergency care? Arch Dis Child 2009;94:888–93.
      OpenUrlAbstract/FREE Full Text
    39. ↵
      1. Buckley DJ,
      2. Curtis PW,
      3. McGirr JG
      . The effect of a general practice after-hours clinic on emergency department presentations: a regression time series analysis. Med J Aust 2010;192:448–51.
      OpenUrlPubMedWeb of Science
    40. ↵
      1. Ryan M,
      2. Spicer M,
      3. Hyett C,
      4. et al
      . Non-urgent presentations to a paediatric emergency department: parental behaviours, expectations and outcomes. Emerg Med Australas 2005;17:457–62.
      OpenUrlCrossRefPubMed
    41. ↵
      1. Redstone P,
      2. Vancura JL,
      3. Barry D,
      4. et al
      . Nonurgent use of the emergency department. J Ambul Care Manage 2008;31:370–6.
      OpenUrlPubMed
    42. ↵
      1. Tang N,
      2. Stein J,
      3. Hsia RY,
      4. et al
      . Trends and characteristics of US emergency department visits, 1997-2007. JAMA 2010;304:664–70.
      OpenUrlCrossRefPubMedWeb of Science
    43. ↵
      1. Afilalo J,
      2. Marinovich A,
      3. Afilalo M,
      4. et al
      . Nonurgent emergency department patient characteristics and barriers to primary care. Acad Emerg Med 2004;11:1302–10.
      OpenUrlCrossRefPubMedWeb of Science
    44. ↵
      1. Rinderknecht AS,
      2. Ho M,
      3. Matykiewicz P,
      4. et al
      . Referral to the emergency department by a primary care provider predicts severity of illness. Pediatrics 2010;126:917–24.
      OpenUrlAbstract/FREE Full Text
    45. ↵
      1. Moll HA
      . Challenges in the validation of triage systems at emergency departments. J Clin Epidemiol 2010;63:384–8.
      OpenUrlCrossRefPubMedWeb of Science

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Study design Prospective observational study.

    • Online appendix Reference classification parameters, matrix and definitions of reference urgency categories.

    • Funding Netherlands Organisation for Health Research and Development (ZonMw) and Erasmus University Medical Centre, Rotterdam, Netherlands. None of the granting agencies listed had any role in the design and conduct of the interpretation of the data or preparation, review or approval of the manuscript.

    • Competing interests None.

    • Ethics approval The ethics approval was provided by the Institutional medical ethics committee; the requirement for informed consent was waived.

    • Provenance and peer review Not commissioned; internally peer reviewed.