Article Text
PDF
Abstract
Introduction Exit block is the most significant cause of poor patient flow and crowding in the emergency department (ED). One proposed strategy to reduce exit block is early admission predictions by triage nurses to allow proactive bed management. We report a systematic review and meta-analysis of the accuracy of nurse prediction of admission at triage.
Methodology We searched MEDLINE, Cochrane, Embase, CINAHL and grey literature, up to and including February 2019. Our criteria were as follows: prospective studies analysing the accuracy of triage nurse intuition—after gathering standard triage information—for predicting disposition for adult ED patients. We analysed the results of this test—nurse prediction of disposition—in a diagnostic test analysis review style, assessing methodology with the Quality Assessment of Diagnostic Accuracy Studies 2 checklist. We generated sensitivity, specificity and likelihood ratios (LRs). We used LRs and pretest probability of admission (baseline admission rate) to find positive and negative post-test probabilities.
Results We reviewed 10 articles. Of these, seven had meta-analysable data (12 282 participants). The studies varied in participant selection and admission rate, but the majority were of moderate quality and exclusion of each in sensitivity analyses made little difference. Sensitivity was 72% and specificity was 83%. Pretest probability of admission was 29%. Positive and negative post-test probabilities of admission were 63% and 12%, respectively.
Conclusion Triage nurse prediction of disposition is not accurate enough to expedite admission for ED patients on a one-to-one basis. Future research should explore the benefit, and best method, of predicting total demand.
- emergency department
- triage
- crowding
- nursing
- emergency departments
- planning
Statistics from Altmetric.com
Key messages
What is already known on this subject
Multiple observational studies have been performed across different settings to determine the accuracy of triage nurse prediction of emergency patient disposition.
Results have been variable across studies.
What this study adds
This paper reports a meta-analysis of observational studies which finds triage nurse prediction of disposition is not sufficiently accurate to aid admission planning on a one-to-one basis.
Introduction
Emergency departments (EDs) worldwide are facing increased pressure to see and treat patients in a timely manner.1 2 An important concern regarding timely patient flow through the ED is exit block, where ED patients needing admission wait for beds to become available in inpatient wards.3 This causes crowding, which adversely affects staff, patients and the system as a whole.4
Patient care in the ED begins with triage and ends with a decision to admit or discharge. An accurate prediction, at triage, of the need for admission might give staff more time to find space for incoming patients, thus reducing exit block.5 As nurses commonly perform triage,6 we sought to answer, ‘Can nurses accurately predict admission at the time of triage?’ Several studies have attempted to answer this with mixed results.5 7 8 We report a systematic review and meta-analysis of this literature, using diagnostic test analysis (DTA) methods.
Analysing nurse prediction using DTA methods
Prediction of disposition is an evaluation of a patient’s clinical condition at triage as being likely or unlikely to need admission. Therefore, it can be considered a diagnostic test. Actual disposition is the gold standard against which we compare prediction. Therefore, this data can be analysed using DTA review methods, allowing determination of sensitivity, specificity and post-test probabilities.
Creation of the structured question
As we have framed this as a DTA review, we defined our question in terms of a modified PICOS:
Participants—adult ED patients.
Index test—triage nurse prediction of disposition (admission or discharge).
Comparison (reference standard) test—actual eventual patient disposition.
Outcome: admission or discharge.
Study design: prospective observational studies.
Potential harms of the test
Bringing forward the bed-finding process involves prioritising beds for hypothetical patients. This is worthwhile if beds are needed.9 Otherwise, it causes unnecessary harm and anxiety to patients who could have received earlier treatment, for example a patient whose elective surgery to remove a tumour has been cancelled.10
Furthermore, triage is highly time pressured.11 Prolonging triage potentially harms the patient being triaged and causes longer waits for newly arriving patients. Asking triage nurses to predict patient disposition on top of their other tasks could only be acceptable if it involved no extra patient evaluation and is minimally time-consuming.
Potential benefits of the test
Maximising bed space allocation efficiency with predictive tools requires fewer resources than increasing hospital capacity. This strategy is particularly relevant now, given the Royal College of Emergency Medicine’s encouragement of increasing the predictive capacity of EDs to better deal with demand, and a rising wave of healthcare innovation aimed at matching capacity to demand.12–14
Methodology
Search strategy
We searched MEDLINE, Cochrane, Embase, CINAHL and grey literature (non-peer-reviewed, eg, a professional body’s report) with Google and OpenGrey, up to and including February 2019, for papers that fit our criteria. A university librarian helped us to develop our strategy. We hand-searched references to ensure that we retrieved all relevant literature. If we identified literature not found in the search, we modified the search and repeated the process. We searched for these concepts: nurses, prediction, disposition and emergency medicine. The online supplementary appendix contains our detailed search strategy.
Supplemental material
We asked the authors of all publications in our meta-analysis if they knew any unpublished literature in this area.
Inclusion and exclusion criteria
Most literature on nurse prediction of disposition concerns adult patients. To increase comparability, we only included studies with adult patients, and excluded disease-specific cohorts. Case studies do not lend themselves to our review. We excluded retrospective studies in anticipation that retrospectively recorded data regarding the patient pathway following triage may have variable interpretations in terms of disposition prediction. We aimed to determine the accuracy of nurse intuition—after gathering standard triage information—in predicting admission without algorithms, to determine its potential as a simple tool to improve flow, so we excluded papers only analysing other methods (eg, scores). Table 1 outlines our criteria.
Inclusion and exclusion criteria
Sensitivity and specificity provide more information than accuracy. Therefore, studies included in the meta-analysis also required true positive, false positive, false negative and true negative data.
For study selection, two researchers scanned all results. Where selection was discrepant, a third adjudicated the appropriateness of including the paper.
Assessment of methodological quality
We assessed methodology using the QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies 2) checklist for appraisal of primary DTA studies. To account for varying methods and quality between studies, we performed sensitivity analyses—meta-analyses without studies whose methods differed from the others. These indicate the summary result’s reliability, and provide alternative, potentially more reliable estimates if they significantly impacted the result.
Data items, extraction and statistical methods
We recorded predicted and final disposition for each study in a 2×2 matrix in Excel. An admission prediction was a positive result and a discharge prediction was negative such that, for example, predicted admission followed by actual admission meant true positive. We used these to calculate sensitivity, specificity and positive and negative predictive values (PPV and NPV).
We consulted a statistician for our meta-analysis. We calculated sensitivity, specificity and 95% CIs for each study using Stata’s ‘diagti’ programme, displayed with a forest plot constructed using Excel. We used Stata’s ‘metandi’ programme to generate an HSROC (Hierarchical Summary Receiver Operator Characteristic) curve which shows how sensitivity and specificity changes at different prediction thresholds, and to calculate summary sensitivity and specificity with the bivariate model. This method uses each study’s sensitivity and specificity to estimate a combined threshold reflecting what has been observed. Therefore, it estimates the likely values which arise from the natural inclination of nurses to predict disposition and current hospital admission protocols. We calculated pretest admission probability as the baseline admission rate across studies. We calculated positive likelihood ratio (LR) as sensitivity: (1-specificity), and negative LR as (1-sensitivity): specificity. Stata’s ‘midas’ programme combined pretest probability and LRs using Fagan’s nomogram to determine positive and negative post-test probabilities (the likelihood of admission after a prediction of admission or discharge, respectively).
Results
We included 10 full texts, shown in figure 1’s Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowchart. No grey literature met our criteria.
PRISMA flowchart of study selection. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
QUADAS-2 methodological analysis
Table 2 contains QUADAS-2 ratings for each study. The online supplementary appendix contains a completed QUADAS-2 form for one paper as an example, and a summary chart for all studies. We commented on methodology concerns in the study characteristics section.
QUADAS-2 assessment for each included paper
Study characteristics
Cameron et al compared triage nurse prediction of admission to the Glasgow Admission Prediction Score (GAPS) in a large teaching hospital in Glasgow, UK.5 We report nurse prediction results; 745/1829 patients were admitted (41%). They excluded 1753 patients in Resuscitation or Minors. They excluded Minors patients on the basis that <2% are admitted and would have readily and accurately been predicted for discharge. Indeed, the admission rate greatly exceeded this review’s average. Nurse experience ranged from band 5 (initial grade) to band 7 (ward manager); predictive ability did not vary by seniority. They derived predictions from a 100-mm Visual Analogue Scale—1 was most likely to be discharged, 100 was most likely to be admitted. The authors assumed any value >50 mm was equivalent to admission prediction which may not have matched outset binary predictions. We anticipate underestimation of specificity and NPV due to the exclusion of Minors patients.
The study of Kosowsky et al took place in a tertiary referral centre in Cincinnati, USA;7 107/521 patients were admitted (21%). They excluded Minors patients. Nurses had at least 6 months of experience. Nurses only made predictions from 12 to 8 pm, when pressure on patient flow was greatest, and presenting complaint types may have differed to other times. Accuracy increased with confidence. The authors failed to report whether decisions to admit were made without knowledge of predictions. However, as such decisions are usually critical to a patient’s care, we felt such knowledge was unlikely to influence disposition. We anticipate underestimation of specificity and NPV due to the exclusion of Minors patients.
Beardsell and Robinson’s study took place in a tertiary referral centre in Southampton, UK;8 491/2190 patients were admitted (22%). They excluded patients who were discharged immediately after triage, redirected to another care provider, self-discharged against medical advice, and those who died in the ED. Nurses had at least 1 year of ED experience. They gave a ‘best guess’ of admission or discharge. We did not anticipate significant bias in any direction.
The study of Stover-Baker et al took place in a level 1 trauma centre in York, USA;15 287/1139 patients were admitted (25%). The patient sample was non-consecutive. They excluded 1105 patients who arrived by ambulance and may have had a higher admission probability. Nurses had at least 1 year of ED experience, completed a triage course and were oriented as trauma nurses. Nurses predicted admission or discharge, level of admission and prediction confidence. On the one hand, excluding ambulance arrivals may reduce sensitivity and PPV. On the other, the non-consecutive sample may introduce selection bias, as confidence may have influenced likelihood of making a prediction, potentially overestimating accuracy with a random sample of patients.
The Emergency Department Prediction of Disposition study (Vaghasiya et al) took place in a tertiary hospital in Sydney, Australia;16 193/795 patients were admitted (24%). While they excluded triage category 1 and 2 patients, this was <4% of the study population and is unlikely to significantly impact the results. Nurse experience level was not stated. Nurses predicted admission, discharge or short stay (staying in ED<24 hours before discharge) which was later converted to discharge, potentially yielding different results to outset binary predictions. We did not anticipate significant bias in any direction.
The study of Alexander et al took place in a tertiary teaching hospital in Hartford, USA; 1304/5135 patients were admitted (25%).17 A convenience sample was used, and nurses could forego prediction if they felt it was inappropriate (637 patients); 261 patients were excluded due to missing admission data. Nurses made binary disposition predictions and predicted level of care if admitted. Nurse experience was not commented on. We anticipate overestimation of predictive accuracy, due to the convenience sample.
The study of Dent took place in a tertiary referral centre in Melbourne, Australia;18 400/673 patients were admitted (59%)—a much higher admission rate than all other studies in this review, suggesting significant methodological discrepancy. They used a convenience sample and attempted to exclude patients with ‘surgical’ complaints, two of whom were included regardless, and mental health presentations. Nursing experience level required was not stated. Nurses made binary disposition predictions and predicted length of stay. The significantly higher admission rate suggests that their exclusions may have left them with a more predictable population than random, overestimating nurse predictive power.
The study of Holdgate et al, which took place in two tertiary referral centres in Liverpool, UK, did not report meta-analysable data. They found that triage nurses could predict adult ED patient disposition with 73% accuracy.19 Predictive accuracy of nurses increased for triage category 1 and 5 patients. Adult admission rate was not stated. Nurses were asked to predict disposition but could forego this in each case. They were more accurate for children than adults. All nurses underwent extra training as well as demonstrating precourse general ED competence. While the ability to forego prediction may suggest overestimated accuracy, admission rates were the same for patients whose disposition was predicted and the others. We therefore anticipated no significant bias in any direction.
Peck et al did not report meta-analysable data but reported the determination coefficient of prediction confidence and final disposition.20 The setting was a tertiary referral centre in Boston, USA. They excluded patients triaged at the bedside. Admission rate was not stated, neither was nurse experience level. Nurses predicted likelihood of admission according to a six-point scale (1 being highly confident in discharge, 6 being highly confident in admission) from 7 am to 5 pm. The r2 was 0.5234, indicating moderate association between confidence and likelihood of admission. We did not anticipate significant bias in any direction.
The study of Noel et al, which took place in two tertiary hospitals and two district general hospitals in Provence-Alpes-Côte d’Azur in France, did not report meta-analysable results;21 2313/9828 patients were admitted (23.5%). Patients who died, left without being seen and self-discharged against medical advice, were excluded. Nurse experience level was not stated. They rated likelihood of admission on a scale of 1–10 (1 means 10% and 10 means 100%), later converted to a binary prediction. However, they only reported accuracy and specificity, not sensitivity, which were 81.4% and 68.8%, respectively. We did not anticipate significant bias in any direction.
Meta-analysis of studies evaluating the sensitivity and specificity of nurses in predicting patient disposition at the point of triage
We performed a meta-analysis on seven studies. Table 3 reports each study’s raw data.
Raw data of each study
Table 4 reports sensitivity, specificity, PPV and NPV for each study, and meta-analysis. Figure 2 shows sensitivity and specificity for each study on a forest plot.
Sensitivity, specificity, positive and negative predictive values for each study and the summary statistics for the meta-analysis
Forest plot of sensitivity and specificity estimates for each study (error bars represent 95% CIs).
The positive post-test probability shown in figure 3 by Fagan’s nomogram (63%) is the PPV and the negative post-test probability (12%) is the NPV of the meta-analysis summary.
Fagan’s nomogram showing pretest probability of admission and positive and negative post-test probabilities of admission. Dotted lines without arrowheads represent 95% Confidence Intervals.
Summary statistics are presented on an HSROC curve with a summary point estimate computed using the bivariate method in figure 4, which shows 72% sensitivity and 83% specificity at the observed threshold. Individual study estimates are near the curve, suggesting comparability between studies, the one furthest being the study of Dent et al which had methodological differences from the others.
Graph showing HSROC curve with combined sensitivity and specificity estimates for all studies included in the meta-analysis. HSROC, Hierarchical Summary Receiver Operator Characteristic.
For sensitivity analyses, we ran the meta-analysis without studies by Cameron et al, Kosowsky et al, Stover-Baker et al, Dent et al, Vaghasiya et al and Alexander et al, which had either methodological or comparability concerns. The results, included in the online supplementary appendix, ranged from 59% to 64% and 10% to 13% for positive and negative post-test probability, respectively.
Discussion
We reviewed 10 studies and performed a meta-analysis on seven of these (12 282 participants). While overall evidence quality was moderate, we identified heterogeneity between studies regarding quality, participant selection and baseline admission rate. Our sensitivity analyses which were based on this had minimal effect on the summary estimates.
Cameron et al and Kosowsky et al’s studies were likely to underestimate accuracy, Alexander et al and Dent et al’s were likely to exaggerate it, while Stover-Baker et al’s had forces in both directions. We surmised that the summary result was more likely to overestimate accuracy, based on the significantly greater number of participants from studies more likely to overestimate accuracy.
Based on this review, the chance of any one ED patient requiring admission was 29%. If a nurse predicts admission, the chance increases to 63%, and if they predict discharge, the chance drops to 12%. If triage nurses were expected to make a prediction for every patient who entered the ED, 37/100 admission predictions would be false positives. This is not sufficiently accurate for bed managers to act on.
Our sensitivity analyses could only exclude one or two concerns at a time (due to the low number of papers); none of our analyses contained entirely comparable data sets. However, no single difference greatly affected the result or led us to believe we had significantly underestimated nurse prediction accuracy.
Conversely, nurses frequently warn colleagues when they are highly confident a patient will need admission. Our review does not question the reliability of these predictions. Indeed, two of the reviewed studies demonstrate predictive accuracy increases with confidence.7 15
A further question arises—could predictions for all patients be aggregated and adjusted for error to provide a real-time impression of total incoming bed demand? This could enable more proactive coordination with other hospitals and activation of escalation policy.11 Adjustment for error could be based on site-specific nurse accuracy data. Neither Cameron et al 5 nor Kosowsky et al 7 found predictive ability varied according to seniority (Kosowsky et al admit they did not assess this prospectively). The other studies did not analyse this. This suggests that error rate may be stable even as nurses change shift. Some prior research, however, has suggested the opposite,22 and such a claim would require more supporting evidence from multiple sites.
We are also aware of scores (eg, GAPS and the Sydney Triage to Admission Risk Tool, also known as START+)5 23 which function identically regardless of staffing changes, and may convey better calibrated aggregate predictions than nurses.
Peck et al suggest that ‘real-time’ prediction can ‘help hospital staff organize, prioritize and adapt their work to manage short-term spikes in demand that exceed long-range average daily demand’.24 Nurse or score-based prediction systems would use live data, providing higher resolution predictions and more confidence to bed managers than historical data alone. Evidence supporting Peck et al.’s suggestion, however, is lacking,25 either because existing protocols and culture do not permit acting on predictions, or because the extra time simply cannot be used when the hospital is full. To determine which is the case, all stakeholders including inpatient staff would need to change practice and culture to enable preparation for predicted incoming demand. Daunting though this may seem in the short-term, having more time to plan and act, and shorter ED stays may well be worth the effort.
However, poor patient flow is a system-wide problem;26 prediction systems would only reach their potential if accompanied by improvements further along the care process.
Peck et al suggest that graded predictions according to level of confidence are more useful than binary predictions for predicting aggregate demand.20 Future studies could attempt similar analyses and would ideally be higher quality than many studies in this review, for example by avoiding convenience samples. We recommend more studies investigate nurse predictions in combination with scores, for example GAPS, as Cameron et al have done.5 Other clinicians’ predictions—such as ED physicians—could also be evaluated, and the role of Artificial Intelligence could be explored in this regard, as Hong et al have done.14 However, future studies in this area could only be helpful if staff attitudes became receptive to the idea of acting based on predictions.
Our review has several limitations. We only undertook a meta-analysis on seven studies and these were not entirely homogeneous. We attempted to increase comparability, and determine the effect of including dissimilar studies, with sensitivity analyses. However, even the most similar studies would have varied in location, nurse experience and admission protocols. We could not isolate the summary statistics for one country due to the low number of included papers. For countries other than the USA, UK and Australia, the results of this review may be less applicable.
Conclusion
Triage nurse prediction of disposition is not accurate enough to help expedite the admission process for patients on a one-to-one basis, if nurses were asked to make such a prediction for all ED patients. More research is required to determine whether predictions of aggregate demand for inpatient beds can help hospital staff prioritise resources in real-time and improve patient flow.
Ethics statements
Acknowledgments
We would like to thank the following people: Professor Khalid Khan MBBS, Barts and the London Medical School, for his guidance on DTA statistics. Jun Xia MSc, Systematic Review Solutions Ltd., Fang Qi MSc, Systematic Review Solutions Ltd. and Professor Mohammad Ansari MD MPhil, University of Ottawa, for their guidance on the correct methods of analysis in DTA reviews. Rebecca Jones BA(Hons) MA, Librarian at Imperial College London, for helping us to develop a comprehensive search strategy.
References
Footnotes
Handling editor Mary Dawood
Contributors MAMA conceived the idea, collected and analysed the data, and wrote most of the paper. He is guarantor. TN and PS contributed to the idea’s conception, collected the data and contributed to writing the paper. TN, PS, HW, FA, CK and AN contributed equally to the paper. HW, FA, CK and AN contributed to the idea’s conception, collected the data and critically revised the manuscript. All coauthors listed above approve submission of this paper to be published.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.