Objective To improve the Manchester Triage System (MTS) in paediatric emergency care.
Methods The authors performed a prospective observational study at the emergency departments of a university and teaching hospital in The Netherlands and included children attending in 2007 and 2008. The authors developed and implemented specific age-dependent modifications for the MTS, based on patient groups where the system's performance was low. Nurses applied the modified system in 11 481 (84%) patients. The reference standard for urgency defined five levels based on a combination of vital signs at presentation, potentially life-threatening conditions, diagnostic resources, therapeutic interventions and follow-up. The reference standard for urgency was previously defined and available in 11 260/11 481 (96%) patients.
Results Compared with the original MTS specificity improved from 79% (95% CI 79% to 80%) to 87% (95% CI 86% to 87%) while sensitivity remained similar ((63%, 95% CI 59% to 66%) vs (64%, 95% CI 60% to 68%)). The diagnostic OR increased (4.1 vs 11).
Conclusions Modifications of the MTS for paediatric emergency care resulted in an improved specificity while sensitivity remained unchanged. Further research should focus on the improvement of sensitivity.
- emergency medical services
- emergency department
- paediatric emergency medicine
Statistics from Altmetric.com
- emergency medical services
- emergency department
- paediatric emergency medicine
Triage is an important tool to manage patient flow safely when clinical need exceeds capacity.1 Several triage systems have been developed and applied in emergency care. The Manchester Triage System (MTS), the Emergency severity Index, the Canadian Triage and acuity scale and the Australasian Triage scale are commonly used triage systems.1–6 They all use specific criteria to triage children.
The MTS consists of 52 flowcharts, which present reasons of encounter, of which 49 are applicable for children. Each flowchart contains discriminators of which selection leads to one out of the five urgency levels. An example of a MTS flowchart is provided in figure 1. In adults, the MTS was shown to be sensitive for early detection of seriously ill patients and for the detection of high-risk chest pain.7–9 The MTS is not valid to detect children with serious bacterial infections.10 The MTS showed a substantial to good inter-rater agreement in adults and children.11 ,12
We studied the validity of the MTS in children and showed a moderate validity with a sensitivity of 63% (95% CI 59% to 66%) and a specificity of 79% (79% to 80%) for identifying high urgency patients. Performance was especially low for patients with a non-traumatic presenting problem, for patients with a young age and for patients with fever.13
The aim of this study was to improve the MTS for paediatric emergency care based on modifications in patient groups in which the MTS performance was low and to evaluate its performance in a new population.
We performed an observational prospective study. Modifications were developed for patient subgroups where the MTS performance was low.13 The modified MTS was implemented at two emergency departments (EDs). We evaluated performance by comparing the MTS urgency levels with a predefined reference standard for urgency. Improvements were evaluated by comparing performance of the modified MTS with performance of the original MTS, as evaluated in our previous study.13
Setting and selection of participants
The Erasmus MC-Sophia Children's hospital in Rotterdam, The Netherlands, is a university hospital with a specific paediatric ED that receives 9000 patient-visits per year. The Haga hospital- Juliana children's Hospital at The Hague, a large urban teaching hospital with a full spectrum of patients, encounters 15 000 paediatric patient-visits per year. Patients aged <16 years were included in the two hospitals, between May 2007–April 2008 and August–December 2007, respectively.
Methods of measurement: MTS
ED nurses performed a short assessment and triaged patients using the MTS. First, the patient's presenting problem will be identified; it leads to the selection of a flowchart. An example of an MTS flowchart is provided in figure 1. MTS flowcharts contain six general discriminators (life threat, pain, haemorrhage, acuteness of onset, level of consciousness and temperature) as well as specific discriminators, which are relevant to the presenting problem. Selection of a discriminator leads to the same urgency level independent of the selected flowchart. The selected discriminator indicates one of the five urgency categories, with a maximum waiting time (‘immediate’ 0 min, ‘very urgent’ 10 min, ‘urgent’ 60 min, ‘standard’ 120 min and ‘non-urgent’ 240 min). Pain is scored on a scale from 0 to 10 and can assign patients to a higher urgency level. Nurses used a digital version of the MTS to triage patients.
The triage nurses were all specialised ED, paediatric nurses. They followed the Manchester Triage Course and the Emergency Nursing Paediatric Course of the Emergency Nursing Association.
Prior to the study, a reference standard was defined based on literature and expert opinion.13 A reference urgency was defined based on a combination of vital signs, diagnosis, diagnostic and therapeutic interventions and hospitalisation/follow-up. It was developed prior to the onset of the study in an expert meeting by paediatricians and a paediatric surgeon.
Patients were considered to be category 1 if their vital signs were deviated according to the limits used in the Pediatric Risk of Mortality Score14 or in case of hyperthermia (temperature >41°C). Patients with hyperthermia have a higher risk for a severe bacterial infection.15 Deviations in temperature, respiratory rate or pulse oximetry and mental status are related to resource use and hospitalisation.16 ,17
Nurses fully examined all children; vital signs were measured at the discretion of the nurse or physician and were not measured if children appeared not ill or had no signs of respiratory or circulatory distress. For those presenting with a medical (non-trauma) problem, temperature was measured in 81%, heart rate in 49% and respiratory rate in 28% of cases. If vital signs were not measured, they were assumed to be normal.
Patients were assigned to category 2 if they had normal vital signs and their presumed diagnosis at the end of their ED consultation was defined as a potential life-threatening condition.13 The potential life-threatening conditions are provided in the online appendix 1. Most of these conditions are associated with a high morbidity and mortality and are discussed in the Advanced Paediatric Life Support workbook as an emergency.18 ,19 Patients presenting with an apparently life-threatening event were suggested to be monitored for 24 h; we therefore considered them as very urgent.20
Patients were allocated to category 3 or 4 depending on the performed diagnostics, administered therapy, hospitalisation and if a follow-up visit was scheduled.
Category 5 was defined if patients did not require any of the resources. Previous studies on other triage systems for paediatric patients showed an association between urgency level and resource use and follow-up. Resource use is associated with the urgency level of the Emergency Severity Index.21–23
A classification matrix of the reference classification and detailed definitions of the reference standard urgencies were published before and are provided in online appendix 1.13 We defined the reference standard for each patient independent of the MTS urgency and based on a computerised application of the classification matrix. We defined overtriage and undertriage when the MTS urgency level was higher and lower, respectively, than the reference standard urgency level.
We studied patient characteristics and their relation to agreement between the MTS and the reference standard urgency distribution.13 Second, modifications were discussed with the ED nurses and paediatricians of both hospitals. The aim of modifying the MTS was to increase correct triage and decrease overtriage without increasing undertriage.
Modifications focused on the patient characteristics: a) age, referral status and presenting problem (traumatic or medical), b) frequently used MTS flowcharts, c) frequently used MTS discriminators and d) combination of a–c. The proportion of number of categories of overtriage and undertriage was calculated for subgroups based on these patient characteristics to evaluate whether attributing a higher or a lower MTS urgency category could result in a better agreement between the MTS and the reference standard urgency. Modifications were created for subgroups if in ≥80% of the patients the reference urgency category was lower than ‘very urgent’.
One new discriminator was inserted after discussion of the modified MTS among the paediatricians working at the participating EDs. For the flowchart ‘Rashes’, a new discriminator ‘Petechiae’ was inserted in the ‘very urgent’ (level 2) category, in order to detect patients with petechiae who are highly suspect of meningococcal septiceamia.
Other modifications were developed using the same method. However, other cut-off levels, as discussed by experts, were used for allocating patients in lower urgency categories. An overview of the modifications with cut-off levels is provided in table 1.
The modified MTS was implemented at the ED of both hospitals. Before implementation, meetings were conducted to instruct the ED nurses on the details of the modifications. A detailed description of the modification was provided. We performed monthly audits on the compliance of the use of the MTS which remained appropriate (76%–96%). Feedback of the compliance was given to the director of the EDs.
We studied reliability by assessing the inter-rater agreement. Twenty-one ED nurses of both studied hospitals classified the urgency level of the same 30 written case scenarios using the modified MTS. The written case scenarios were obtained and selected from other studies on reliability of triage systems for children24 ,25 and six local scenarios were added. We aimed to evenly distribute age, presenting problems and acuity level (according to the Emergency Severity Index). The selected urgency level per case scenario was compared between the nurses. We used the intraclass correlation coefficient as a measure of inter-rater agreement, which can be used for multiple raters and can be interpreted as quadratic weighted κ, as was used before to study reliability of the original MTS in children.12 The inter-rater agreement in acuity level (quadratic weighted κ) 0.77 (95% CI 0.67 to 0.86) was similar to the reliability of the original MTS (0.83, 95% CI 0.74 to 0.91).12
Patient characteristics, selected MTS flowchart, MTS discriminator and the urgency category were recorded in the computerised triage system. Urgency levels according to the original MTS could be determined by the selected MTS flowchart and discriminator. Nurses or physicians recorded data concerning vital signs, diagnosis, diagnostic and therapeutic interventions, hospitalisation and follow-up on structured electronic ED forms. Trained medical students gathered and entered the data for the reference standard, independent of the triage outcome, using SPSS-Data Entry V.4. The database was checked for consistency and outliers. Data on laboratory tests were obtained from the hospital information system.
Primary data analysis
To assess performance of the modified MTS we calculated percentage over-, under and correct triage. Second, we calculated sensitivity, specificity and the diagnostic OR (DOR). Patients were categorised as high urgent (level 1 or 2) or low urgent (level 3, 4 or 5). The DOR describes the odds of an MTS high urgency in high urgency patients, according to the reference standard, as compared with the odds of an MTS high urgency in low urgency patients, according to the reference standard (DOR=(sensitivity/1-sensitivity)/(1-specificity/specificity)).26
A risk stratification table shows the extent to which high urgency patients, according to the reference standard, are assigned to MTS high urgency categories. Second, it shows if low urgency patients, according to the reference standard, are assigned to MTS low urgency categories27 (table 2). SPSS V.15.0 was used for statistical analysis.
Characteristics of study subjects
In total, 13 654 patients presented at the ED of the Sophia Children's Hospital and Juliana Children's Hospital during the study period. In all, 11 927 (84%) were triaged using the modified MTS. In the Sophia Children's hospital, the nurses had the possibility to over-rule the system, and did so in 3.4% (221/6531) of the patients; these patients were excluded. The reference standard for urgency could be determined in 11 260 patients (96%).
The median age was 3.4 years (IQR 1.2–8.2), 4748 (42%) were female subjects and 5112 (45%) were referred by a healthcare professional. Patients were triaged using the flowcharts ‘Shortness of breath in children’ in 13.9%, ‘General’ in 11.2%, ‘Vomiting or diarrhoea’ in 8.7%, ‘Worried parent’ in 7.8%, ‘Abdominal pain in children’ in 5.9%, ‘Rashes’ in 2.7%, ‘Fits’ in 2.7%, ‘Urinary problems’ in 2.4%, ‘Ear problems’ in 1.7% and 13.7% of the patients were triaged with other medical flowcharts. Flowcharts for traumatic problems were applied in 29.3%. These characteristics were comparable with patients included in the first dataset.13
The urgency levels of patients triaged using the modified MTS were compared with the five reference standard urgency levels. The modified MTS agreed in 37% (n=4204) with the reference standard urgency, 36% (n=4091) were overtriaged by one category and 11% (n=1276) by more than one category. About 13% (n=1477) were undertriaged by one category and 2% (n=212) by >1 category.
Sensitivity was 64% (95% CI 60% to 68%) and specificity 87% (95% CI 86% to 87%), resulting in a DOR of 11.5 (95% CI 9.6 to 14) (table 3).
Due to the modifications, 930 patients (8.2%) were reclassified to other urgency categories. For the three highest original MTS urgency categories, all reclassified patients were allocated to a lower urgency category. The reclassified patients who were allocated to a lower urgency category had a lower incidence of high urgency cases according to the reference standard, compared with patients who were not reclassified (2.2% vs 5.7%). Table 2 provides the reclassification of patients from the original MTS urgency to the modified MTS urgency per urgency level.
This study shows that some small modifications resulted in an improved version of the MTS in paediatric emergency care. Specificity improved and sensitivity remained similar. The DOR improved. The modifications reclassified 8.2% of the patients.
Misclassification may partly be explained by incorrect application of the system. Therefore, we checked the reliability of the modified MTS. The inter-rater agreement was similar when compared with the inter-rater agreement of the original MTS (Kw 0.77, 95% CI 0.67 to 0.86) indicating that the modified MTS was correctly applied in most cases.
Since triage systems are usually developed by expert opinion, it is important to evaluate its performance and modify the system if performance is not optimal.
We identified patient groups where the MTS has a low performance for children and developed modifications based on and for these patient groups. The modifications were implemented and evaluated in a new population in two ED settings. As was shown for clinical prediction models, even if a model is derived from patient data (instead of expert opinion), it is important to validate it in a new population since generalisability of models with good internal validation measures can be very disappointing. This can be caused by inadequate development of the model or major differences between the populations in which the model was developed and validated.28 Our validation set was comparable with the first dataset (2006–2007) on age, gender, flowchart distribution and reference standard urgency classification. Therefore, we can assume that improvement in performance is mostly due to the modification and less to a change in population characteristics.
The Manchester triage group developed modifications in 2006 to improve the system for children and adults.4 These modifications were developed by comments from users and were not validated.
Specific modifications for children were developed in other common used triage systems in paediatric emergency care as well. To our knowledge, only one study evaluated the effects of one of these modifications. For the paediatric version of the level 5 Canadian Triage and Acuity Scale,3 children with fever were assigned to a lower urgency category (urgency category 4 instead of urgency category 3) if they had no signs of toxicity (defined as unexplained crying before examination, difficulty awakening or poor response to the physical evaluation) and were older than 6 months of age. Admission rates were compared between patients who remained in level 3 and the modified patients. The modified patients had lower admission rates and none died or required Intensive Care Unit admission.29
Other studies evaluated performance of triage systems in children and employed resource use, hospitalisation, length of stay or Intensive Care Unit admission as outcome.21 ,22 ,30 ,31 However, these studies did not aim to identify specific patient groups in which the studied triage system performed less. Two studies evaluated performance of triage systems in adults. They identified patient groups based on age, sex, presenting problem and vital signs, in which performance was low, in order to develop modifications that could improve performance of triage systems.32 ,33
After modification, the performance of the MTS improved. The modified MTS showed a specificity of 87% and a sensitivity of 64%. Users should realise that although the high specificity indicates that the system has a good ability to identify low urgent patients, the sensitivity is only moderate.
Although we improved the systems, further improvement in performance, especially sensitivity, should be developed. Since the MTS uses descriptions of vitally threatened patients instead of concrete limits for vital signs to be concerned as deviated, some high urgency patients will be missed. For example, patients with ‘shock’ are defined as level 1 (‘immediate’), patients with a tachycardia, arrhythmia or deviated blood pressure without shock are not considered as high urgent, unless other high urgent discriminators are present.4 Our previous study showed that severe undertriage was rare using the MTS; however, 89% of undertriaged children had abnormal vital signs, which the MTS had not recognised.34
Large datasets are necessary to identify patient characteristics and discriminators which predict urgent with a high discriminative value.
In the studied population, 45% were referred by a general practitioner. This proportion is comparable with referral rates in other settings where primary care physicians are available, with referral rates ranging from 30% to 80% (European countries,35–38 Australia,39 ,40 the USA41 ,42 and Canada43 ,44). Therefore, we think that are results are generalisable to these other populations as well.
We developed a proxy for urgency which was determined out of characteristics of the patients' ED consultation (vital signs, life-threatening conditions, performed diagnostics and therapy, scheduled follow-up or hospitalisation). The individual items predict severity of disease and were combined by expert opinion to define a reference standard for urgency (five levels). Although it is not a real ‘gold standard’, it is a proxy for ‘real urgency’.13 ,45 No other outcome measure for urgency that defines five urgency levels is previously described in the literature.
Modifications were developed after applying the original MTS in 17 600 patients in 2006–2007 who visited the ED of a large teaching hospital or a university children's hospital in The Netherlands. We aimed to develop modifications resulting in an optimal performance of the MTS and to minimise undertriage and overtriage. Therefore, we had to develop modified discriminators which allocate a patient to different urgency levels if different flowcharts are used. (For example, a child aged between 2 months and 3 years with fever will have a ‘Urgent’ triage level when triaged using the flowchart ‘Worried parent’, ‘General’ or ‘Urinary problems’ while a ‘Standard’ triage level is attributed when the child is triaged using the flowchart ‘Sore throat’, ‘Ear problems’ or ‘Rashes’.) If a computerised system is used, these modifications can easily be implemented.
Evaluation of performance of the modifications was done in a new population. The modified MTS was applied in 13 654 patients in 2007–2008. Therefore, our results are representative for a wide population of children presenting at the ED. However, since the modified MTS was applied in the same setting in which the modifications were developed, the modifications should be validated in different settings as well.
Modifications of the MTS for paediatric emergency care resulted in an improved specificity while sensitivity remained similar. The DOR also improved substantially. The modified MTS should be validated in different settings to further evaluate its performance. In all, 8.2% of the cases were reclassified according to the modified MTS; therefore, extended data collections are necessary to further improve the MTS in paediatric emergency care.
We thank the ED nurses of the Erasmus MC-Sophia children's hospital and the Haga Hospital-Juliana Children's hospital for their cooperation in this study, Marcel de Wilde, BSc, Department of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands, for technical support. Furthermore, we thank Csila de Knijff, medical student, for her participation in the data collection.
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Download Supplementary Data (PDF) - Manuscript file of format pdf
Study design Prospective observational study.
Online appendix Reference classification parameters, matrix and definitions of reference urgency categories.
Funding Netherlands Organisation for Health Research and Development (ZonMw) and Erasmus University Medical Centre, Rotterdam, Netherlands. None of the granting agencies listed had any role in the design and conduct of the interpretation of the data or preparation, review or approval of the manuscript.
Competing interests None.
Ethics approval The ethics approval was provided by the Institutional medical ethics committee; the requirement for informed consent was waived.
Provenance and peer review Not commissioned; internally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.