Article Text
PDF
Abstract
Background It is postulated that early determination of the need for admission can improve flow through EDs. There are several scoring systems which have been developed for predicting patient admission at triage, although they have not been directly compared. In addition, it is not known if these scoring systems perform better than clinical judgement. Therefore, the aim of this study was to validate existing tools in predicting hospital admission during triage and then compare them with the clinical judgement of triage nurses.
Methods To conduct this prospective, single-centre observational study, we enrolled consecutive adult patients who presented between 30 September 2019 and 25 October 2019 at the ED of a large teaching hospital in Milan, Italy. For each patient, triage nurses recorded all of the variables needed to perform Ambulatory (AMB), Glasgow Admission Prediction (GAP) and Sydney Triage to Admission Risk Tool (START) scoring. The probability of admission was estimated by the triage nurses using clinical judgement and expressed as a percentage from 0 to 100 with intervals of 5. Nurse estimates were dichotomised for analysis, with ≥50% likelihood being a prediction of admission. Receiver operating characteristic curves were generated for accuracy of the predictions. Area under the curve (AUC) with 95% CI for each of the scores and for the nursing judgements was also calculated.
Results A total of 1710 patients (844 men; median age, 54 years (IQR: 34–75)) and 35 nurses (15 men; median age, 37 years (IQR: 33–48)) were included in this study. Among these patients, 310 (18%) were admitted to hospital from the ED. AUC values for AMB, GAP and START scores were 0.77 (95% CI: 0.74 to 0.79), 0.72 (95% CI: 0.69 to 0.75) and 0.61 (95% CI: 0.58 to 0.64), respectively. The AUC for nurse clinical judgement was 0.86 (95% CI: 0.84 to 0.89).
Conclusion AMB, GAP and START scores provided moderate accuracy in predicting patient admission. However, all of the scores were significantly worse than the clinical judgement of the triage nurses.
- triage
- emergency nursing
- emergency department
- hospitalisations
Data availability statement
Data are available upon reasonable request. All data relevant to the study are included in the article or uploaded as supplemental information. All the de-identified participant data and the statistical analysis plan will be available to researchers who will provide a methodologically sound proposal to the corresponding author.
Statistics from Altmetric.com
Key messages
What is already known on this subject
Several scoring systems have been developed to predict admission at triage with the idea that this could optimise flow within the ED. Previous studies of triage nurses’ ability and the use of scores to predict admission have shown promising results but these prediction methods have not been compared simultaneously.
What this study adds
In this single-centre study, the triage nurses exhibited an excellent ability to predict admission, and were more accurate than Ambulatory, Glasgow Admission Prediction and Sydney Triage to Admission Risk Toolscores. This may suggest new approaches to admission prediction and the flow of ED patients.
Introduction
ED crowding leads to an increase in waiting time, adverse events, work-related stress for healthcare professionals, an increased likelihood of patients leaving the ED before or after their medical evaluation, and a reduction in perceived quality of hospital care.1 2 Evidence suggests that a reduction in ED crowding is associated with better clinical outcomes.3 Several solutions such as fast track units, observation units and lean methodology in process design have been proposed to improve crowding in the ED.4–6
A clinician’s final decisions regarding discharge or admission require considerable time.7 Accumulating evidence has suggested that rapid and early recognition of patients who are likely to be admitted to hospital might be incorporated in the patients’ management, optimising the allocation of resources and improving patient flow.4 6 8 In addition, informing patients and their family members regarding the probability of admission early in the ED visit allows them to organise their work and personal commitments more effectively.9 Studies implementing lean methodology, reallocating the medical and nursing staff based on likelihood of admission or discharge showed an improvement in patients’ flow.6
Various tools have been developed to predict admission during triage, yet none of them have been widely adopted.8 10–14 Ambulatory (AMB), Glasgow Admission Prediction (GAP) and Sydney Triage to Admission Risk Tool (START)9 15 16 have exhibited good predictive accuracy in derivation and validation studies.10 17 The subjective judgement of triage nurses in predicting admission has shown promising results but a recent systematic review questioned the accuracy of this method.18–22 To date, the available scoring systems have not been simultaneously compared with each other or to clinical judgement for the same cohort. Therefore, the aim of this study was to compare the existing tools and the clinical judgement of triage nurses in predicting hospital admission.
Methods
We conducted a prospective, single-centre observational study at the general ED of Fondazione IRCCS Ca’ Granda Ospedale Maggiore Policlinico (Milan, Italy) between 30 September 2019 and 25 October 2019. The hospital is a tertiary teaching hospital with 70 000 adult ED visits annually. The gynaecological and paediatric EDs are in other buildings of the same hospital with separate staff. The staff of the general ED consists of 12 nurses (2 of these are in triage), 6 staff physicians, 2 emergency residents and 3 nursing assistants during daytime shifts and 9 nurses (2 in triage), 3 staff physicians, 1 emergency resident and 2 nursing assistants during night shifts. In order to be eligible to perform triage, nurses need at least 6 months’ experience in the ED, attend a triage course and undergo a period of coaching with an experienced triage nurse.
In Italy, three different triage models are in use. Our ED uses a ‘biphasic model’: one nurse first assesses patients and assigns a priority code based on the leading symptom; a second nurse reassesses the patient, may change priority code if needed, performs first-level tests (eg, blood tests and ECG) and administers pain medicines based on standardised protocols. The triage scoring system uses four codes: red (critical patient requiring immediate access to treatment area), yellow (urgent with possible life-threatening condition), green (non-critical, deferrable condition), white (non-critical, non-urgent condition). ED physicians decide disposition and allocate patients to available hospital beds in collaboration with a bed manager nurse.
Population
Nurse recruitment
We enrolled all nurses who qualified for triage function. For each nurse, we recorded their age, gender, years of experience as a nurse, years of experience in the ED and years of experience performing triage. Each enrolled nurse provided written informed consent prior to participation in this study.
Patient recruitment
We enrolled consecutive adult patients (≥18 years old) who were admitted alive to the general ED during the study period. Patients enrolled in ‘fast track’ paths for specialist visits (ie, otolaryngology, ophthalmology, dermatology, orthopaedics), as well as patients of the paediatric and gynaecological EDs, were excluded as they are not seen in our ED.
Scoring systems
The AMB scoring system was developed in South Wales with the aim of identifying patients likely to have early discharges (within 12 hours of assessment and treatment in hospital).9 AMB scoring ranges from −1 (indicating a low probability of early discharge) to 8 (indicating a high probability for early discharge). In the derivation study, an AMB score cut-off between 4 and 5 gave the most favourable sensitivity (96%) and specificity (65%). Patients receiving a score ≥5 are considered likely to be discharged.
GAP scoring was developed in North Glasgow to estimate the probability of admission.16 The derivation study found that a ‘high probability’ score of >25 identified over one-third of admissions immediately; a score of <8 would identify over half of all discharges at triage. When used as a binary predictor, the optimum cut-off of >15 points had a sensitivity of 78% and a specificity of 81.7% for predicting admission.
START scoring, developed in New South Wales.15 uses deciles of risk scores with corresponding mean predicted probabilities of admission: risk score <1 (3%), 1–10 (14%), 11–20 (47%), 20–30 (81%), 30–40 (96%), >40 (99%). The optimal cut-off identified in the derivation study was a risk score of 13. Therefore, patients scoring ≥14 are considered to have a high probability of being admitted.
The items of the three scoring systems are reported in table 1. Details of the items used for assigning the scores and the points assigned to each item are provided in the online supplemental appendix.
Supplemental material
Items included in the AMB, GAP and START scoring systems
The GAP and START scores include the assigned triage acuity, based on a five-level system. As we use a four-level triage system, for GAP we considered red codes as triage category 1 (20 points), yellow codes as triage category 2 or 3+ (10 points), green codes as triage category 3 (5 points), white codes as triage category 4 or 5 (0 points). For START, red codes were equivalent to triage category 1 (24 points), yellow codes as triage category 2 (16 points), green codes as triage category 3 (11 points), white codes as triage category 4 (5 points). No patient was considered to have a triage category 5 for START (0 points).
Data collection
Prior to data collection, information meetings to explain the study design and aim show the data collection and consent acquisition process was held with the nurses participating in this study. For each patient, we asked the triage nurse to: (1) collect the variables needed to calculate AMB, GAP and START scores; and (2) estimate the probability of admission according to their clinical judgement as a percentage from 0 to 100 with intervals of 5. The nurses who first assessed the patients recorded their clinical judgement immediately after triage in a survey created on the ‘SURVIO’ (Survio, Czech Republic), accessible from the two computers at triage.
The nurses were not asked to calculate the scores, but only to record the variables to allow the investigators to calculate them later. Triage nurses were essentially blinded to total scores predicting admission because they were not routinely used in the ED and the scores were not presented during meetings. It was also requested that the nurses not research any of the scores before the end of the data collection period in order to remain blinded.
Patient outcome
The primary outcome was hospital admission. Patients who died during observation in the ED, and those who refused admission, or were transferred to other hospitals when beds were not available, were considered to be admitted. After recruitment, we excluded from the analysis patients who voluntarily left before or after their ED medical evaluation was completed because a disposition was not available.
Sample size
Assuming a 19% admission rate (based on 2018 data), and considering a reported sensitivity of approximately 85% for the predictive tools employed in previous studies,9–11 15 17 we estimated that approximately 2000 patients were needed in this study to achieve 85% sensitivity with acceptable precision (ie, 95% CI: 81 to 88).
Statistical analysis
Characteristics of the participating nurses and patients were described by using median and IQR values for continuous variables, and frequencies with proportions for categorical variables. Χ2 tests and t-tests were applied to assess differences between enrolled and non-enrolled patients, and between admitted and discharged patients, for categorical and continuous variables, respectively. To determine the diagnostic accuracy measures, we used the cut-offs of the derivation studies for the scores and dichotomised clinical judgement using a cut-off value according to the maximum Youden Index (J).23 This meant that for nursing judgement, a probable ≥50% was a prediction of admission. For each probability estimated by a nurse’s judgement and for each result of the scores, sensitivity, specificity and predictive values with 95% CI were calculated. Receiver operating characteristic (ROC) curves were generated and area under the curve (AUC) values were calculated with 95% CI for each prediction method.
We also performed an additional analysis comparing sensitivity, specificity and predictive values of the scores and clinical judgement using the cut-off value according to the maximum Youden Index for prediction methods based on our data.
To assess the possible confounding role that experience of a triage nurse could have on the ability to predict admission, a logistic regression analysis was planned. Possible interactions between experience and probability of admission were examined. Experience was defined based on age, years of experience as a nurse, years as an ED nurse and years performing triage.
We considered p values of <0.05 (two tailed) to be statistically significant. All statistical analyses were performed with SAS statistical software (V.9.4, SAS Institute).
Patient and public involvement
This research was done without patient involvement.
Results
A total of 40 nurses met the inclusion criteria for this study. However, one nurse declined to participate and four nurses did not perform triage during the study period. Thus, 35 nurses participated in this study. Among those enrolled, 43% were male and the median age was 37 years (IQR: 33–48). The median experience as a nurse was 12 years (IQR: 8–21), median experience in the ED was 9 years (IQR: 5–19) and median experience performing triage was 5 years (IQR: 1–16). Each nurse enrolled a median of 34 patients (IQR: 16–63).
During the study period, 4036 patients arrived at the general ED of Fondazione IRCCS Ca’ Granda Ospedale Maggiore Policlinico and 3404 patients met the inclusion criteria for this study (figure 1). Of these, 1373 patients were not recruited due to organisational reasons (ie, the nurse did not participate in the study, information technology problems, lack of time for recruitment because of absence of the second triage nurse due to needs in other ED areas). After the recruitment phase, we excluded 148 patients for errors in completing forms during data collection, and 173 patients who left the ED before a final disposition, leaving 1710 in the study cohort.
Study flow diagram.
Table 2 presents the characteristics of the 1710 included patients and the characteristics of the 1373 who were not enrolled. Median age of the study population was 54 years; the group was roughly evenly divided between men and women. The largest triage category was green, and the majority had a new early warning score of ≤5. Somewhat more patients arrived in daytime hours. Patients who were not analysed due to lack of recruitment were similar to the study group, except for having a lower median age (51 years (IQR: 32–72)) and a higher prevalence of white triage codes (3%).
Characteristics of the enrolled and non-enrolled patients
Among the study patients, 310 (18.1%) were admitted to the hospital. Demographic characteristics, median AMB, GAP and START scores, and probability of admission according to the nurses’ clinical judgement of admitted and discharged patients are reported in table 3.
Characteristics of the admitted and discharged patients
Figure 2 presents ROC curves for the AMB, GAP and START scores, and for the nurses’ clinical judgements. AUC for nurse clinical judgement is significantly higher than the AUC of the scores analysed (p<0.0001 for all comparisons).
Comparison of receiver operating characteristic curves. AMB, Ambulatory; GAP, Glasgow Admission Prediction; START, Sydney Triage to Admission Risk Tool.
Sensitivity, specificity and predictive values of the scores at the cut-offs used in the derivation studies and of clinical judgement are reported in table 4.
Sensitivity, specificity, and positive (PPVs) and negative predictive values (NPVs) of AMB, GAP, and START scores at the cut-offs used in the derivation studies and of clinical judgement at the cut-off with maximum Youden Index (>50% probability of admission)
Sensitivity, specificity and predictive values of the scores at the optimum cut-off for our data (using maximum Youden Index) and nurses’ judgements are reported in the online supplemental appendix. There was no significant difference between the cut-off using our data and the cut-off of the original derivation studies for all the scoring systems.
In bivariate analyses, age, years of experience as a nurse, years of experience as an ED nurse and years of performing triage functions were not found to be related to a nurse’s ability to predict admission. Thus, regression analysis was not performed.
Discussion
The results of the present study demonstrate that the clinical judgement of triage nurses, regardless of their experience, was very good at predicting hospital admission, and the accuracy of their clinical judgement was higher than the AMB, GAP and START scoring systems.
The accuracy of triage nurses’ clinical judgement in predicting hospital admission has previously been assessed. For example, in the study conducted by Stover-Baker et al,20 triage nurses were asked after the triage phase to express ‘yes’ or ‘no’ if they believed that a patient would be admitted. The sensitivity and specificity values for predicting admission were 76% and 85%, respectively. Alexander et al 21 reported the sensitivity and specificity for prediction of admission were 71.5% and 88%, respectively. A recent systematic review and meta-analysis of the accuracy of nurse prediction of admission at triage showed a sensitivity and specificity of 72% and 83%, respectively.22 The present results confirm the findings of these previous studies. When adopting a cut-off value of ≥50% probability of admission, the sensitivity and specificity values were 76.5% and 84.5%, respectively.
The AUC of the triage nurses’ clinical judgement in the present study was 0.86 (95% CI: 0.84 to 0.89). A study by Cameron et al which compared the accuracy of triage nurses and the GAP score11 found similar predictive accuracy, with AUC values of 0.876 (95% CI: 0.860 to 0.892) and 0.875 (95% CI: 0.859 to 0.891), respectively. The sensitivity and specificity values for clinical judgement were 81.2% and 77.4%, respectively. In a study comparing GAP score with AMB, GAP (AUC: 0.807; 95% CI: 0.785 to 0.830) was found to be superior to AMB (AUC: 0.743; 95% CI: 0.717 to 0.769).10 In contrast, we observed a slightly better predictive accuracy for AMB scoring (AUC: 0.77; 95% CI: 0.74 to 0.79) over both GAP scoring (AUC: 0.72; 95% CI: 0.69 to 0.75) and START scoring (AUC: 0.61; 95% CI: 0.58 to 0.64) in the present study.
To our knowledge, this is the first study to directly compare all of the currently available tools for predicting hospital admission from triage. All three sets of scores exhibited better predictive accuracy in previously performed derivation and internal validation studies than in the present study.9 15–17 This is consistent with other validation studies of these scores.10 12 13 While predictive accuracy in validation studies is often lower than in the derivation studies, this may also be due to the difficulty associated with adapting the peculiarities of some scoring systems to different contexts. For example, in the UK where GAP was developed, the two score items ‘being sent to the ED by a general practitioner (GP)’ and ‘arriving at the ED by ambulance’ are usually indicative of a higher need of in-hospital treatment because patients have already been seen by their GP or determination was made by the ambulance service to transport to the ED. In Italy, many patients arrive at the ED without seeing a GP, and emergency medical services cannot refuse ambulance transportation to patients.10 GAP also uses a priority code system adopted in the UK, while START uses a priority code system adopted in Australia. Both scores are based on a five-level triage severity system. The scoring therefore needed to be adapted to a four-colour code system used in Italy. Furthermore, START scoring requires categorising the various reasons of presentations into 1 of the 18 main symptoms which may be difficult or result in different categorisations in another country.15
What may be of greatest interest is that, while clinical judgement could be subjective and not entirely reproducible, yet its predictive accuracy is very good in all of the contexts in which it has been studied, including our own.18–21
The results of the present study support the excellent ability of triage nurses to predict hospital admission over clinical prediction tools. Consequently, we would advocate that rather than search for more reliable objective scores, it may be more efficient to focus on how to incorporate an early judgement on disposition into the flow of ED patients.13 Kelly et al used lean methodology to reallocate medical and nursing staff based on the likelihood of admission or discharge,6 and based on the theory that the two patient groups require different levels of observation, intensity of investigation and treatment, consultations and organisation of home support or follow-up. By differentiating those likely to be admitted from those not, the authors found reductions in episodes of ambulance diversion and reduced/stable waiting times, with no adverse events despite an increase in workload.
While studies of GAP and AMB and nursing judgement have looked at accuracy of predictions, they have not reported using admission prediction to actually improve crowding. Thus, its effect is uncertain. In current practice, bed requests and preparation are delayed until admission is certain. Early prediction of admission might allow bed managers to start planning for bed availability long before the physician places the order to admit, which could possibly shorten the boarding time. Peck et al suggest that real-time prediction can help hospital staff organise and mobilise resources before crowding becomes an issue.24 Therefore, future studies should assess if management strategies based on an early identification of patients needing admission improve patient outcomes or flows.
Limitations
The study was carried out in a single hospital with a limited number of nurses who performed the triage However, a similar number of nurses have been reported in other single-centre studies.20 21 Second, even though the nurses declared ignorance of the scoring systems evaluated in this study, it is possible that some of them became aware of the tools’ predicting abilities, with a resulting influence on their clinical judgement. Third, while we enrolled a large number of patients, some were not enrolled for organisational reasons, or because they left the ED before or after their medical evaluation. This could have introduced bias into our study. However, the characteristics of the patients are similar to other studies and the characteristics of the enrolled versus non-enrolled patients are comparable. Moreover, the proportion of patients leaving the ED without a full evaluation is consistent with the proportion observed in previous months. Finally, the performance of the scores may have been influenced by the need to adapt the scoring for a four-tier triage system, and the difficulties associated with assessing some of the predictors developed in other countries. However, if the scores are not accurate due to problems with their application and interpretation in other healthcare systems, then their usefulness is limited in clinical practice.
Conclusions
Triage nurses exhibited an excellent ability to predict admission compared with the AMB, GAP and START scores assigned to the cohort of patients examined. While the scores showed a fair predictive capacity, they were not as accurate as the clinical judgement of the triage nurses.
Data availability statement
Data are available upon reasonable request. All data relevant to the study are included in the article or uploaded as supplemental information. All the de-identified participant data and the statistical analysis plan will be available to researchers who will provide a methodologically sound proposal to the corresponding author.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and was approved by the Institutional Review Board of Milano Area (approval no. 755_2019bis, 18 September 2019). Participants gave informed consent to participate in the study before taking part.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Handling editor Ellen J Weber
Contributors MSa, MSo, GCa and GCo designed the study. MSa, JC and FB collected the data. MSa, MSo and GCa analysed and interpreted the data. MSo, MSa and GCo drafted the manuscript. All the authors critically revised the manuscript for important intellectual content. GCa provided statistical expertise. MSa is responsible for the overall content as guarantor.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.