http://orcid.org/0000-0001-9072-2469
Figures
Abstract
Objective
In hospitalized patients, the risk of sepsis-related mortality can be assessed using the quick Sepsis-related Organ Failure Assessment (qSOFA). Currently, different tools that predict deterioration such as the National Early Warning Score (NEWS) have been introduced in clinical practice in Emergency Departments (ED) worldwide. It remains ambiguous which screening tool for mortality at the ED is best. The objective of this study was to evaluate the predictive performance for mortality of two sepsis-based scores (i.e. qSOFA and Systemic Inflammatory Response Syndrome (SIRS)-criteria) compared to the more general NEWS score, in patients with suspected infection directly at presentation to the ED.
Methods
We performed a retrospective cohort study. Patients who presented to the ED between June 2012 and May 2016 with suspected sepsis in a large tertiary care center were included. Suspected sepsis was defined as initiation of intravenous antibiotics and/or collection of any culture in the ED. Outcome was defined as 10-day and 30-day mortality after ED presentation. Predictive performance was expressed as discrimination (AUC) and calibration using Hosmer-Lemeshow goodness-of-fit test. Subsequently, sensitivity, and specificity were calculated.
Results
In total 8,204 patients were included of whom 286 (3.5%) died within ten days and 490 (6.0%) within 30 days after presentation. NEWS had the best performance, followed by qSOFA and SIRS (10-day AUC: 0.837, 0.744, 0.646, 30-day AUC: 0.779, 0.697, 0.631). qSOFA (≥2) lacked a high sensitivity versus SIRS (≥2) and NEWS (≥7) (28.5%, 77.2%, 68.0%), whilst entailing highest specificity versus NEWS and SIRS (93.7%, 66.5%, 37.6%).
Citation: Brink A, Alsma J, Verdonschot RJCG, Rood PPM, Zietse R, Lingsma HF, et al. (2019) Predicting mortality in patients with suspected sepsis at the Emergency Department; A retrospective cohort study comparing qSOFA, SIRS and National Early Warning Score. PLoS ONE 14(1): e0211133. https://doi.org/10.1371/journal.pone.0211133
Editor: Juan Carlos Lopez-Delgado, Hospital Universitari Bellvitge, SPAIN
Received: July 19, 2018; Accepted: January 8, 2019; Published: January 25, 2019
Copyright: © 2019 Brink et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data are available from the RePub, Erasmus University Repository of the Erasmus University Medical Center, Rotterdam (http://hdl.handle.net/1765/113273). The Erasmus University Medical Center has a restrictive policy in sharing de-identified data on third party servers, which is a result of the General Data Protection Regulation.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: AUC, Area Under the Curve; AVPU, Level of consciousness scale Alert, Verbal, Pain, Unresponsive; ED, Emergency Department; GCS, Glasgow Coma Scale; HR, Heart Rate; ICU, Intensive care unit; IQR, Interquartile range; LR, Likelihood Ratio; MTS, Manchester Triage System; NEWS, National Early Warning Score; qSOFA, quick Sepsis-related Organ Failure Assessment; SBP, Systolic Blood pressure; SD, Standard deviation; SIRS, Systemic Inflammatory Response Syndrome; SOFA, Sepsis-related Organ Failure Assessment; Χ2, Chi-squared
Introduction
Sepsis is a syndrome characterised by both signs of infection and manifestations of a systemic host response [1]. Sepsis is the primary cause of mortality from infection. The definition of sepsis has changed throughout the last decades. In February 2016 the Third International Consensus Definition for Sepsis (Sepsis-3) replaced the Sepsis-2 definition dating from 2001 [1–3]. Sepsis is currently defined as a “life-threatening organ dysfunction caused by a dysregulated host response to infection”, in which organ dysfunction is represented by an increase of at least two points in the Sequential Organ Failure Assessment (SOFA) score [1]. The Systemic Inflammatory Response Syndrome (SIRS) score, which was part of the definition in Sepsis-1 and -2, has been abandoned.
The quick Sepsis-related Organ Failure Assessment (qSOFA) was introduced with the new Sepsis-3 definition [4]. However, not all medical societies support this new definition[5, 6]. The qSOFA consists of three parameters (i.e. low systolic blood pressure (≤100 mmHg), tachypnea (≥22 /minute) and altered mental status (Glasgow Coma Scale (GCS) <15 / AVPU<Alert)), with a maximum score of three points. qSOFA is a bedside prompt to identify patients with a suspected infection who are at greater risk for a poor outcome. It is a simplified score based on the SOFA score. Early identification of these patients potentially results in earlier adequate treatment and a decrease in mortality. qSOFA aims to prognosticate the course of sepsis and intends to predict sepsis-related mortality and adverse events; a score of two points or higher gives a three to 14-fold increase in in-hospital mortality [4]. The qSOFA score is claimed to be more accurate than SOFA in departments outside the intensive care unit (ICU), however the use of qSOFA in the Emergency Department (ED) is questionable [4, 7–10]. The authors of Sepsis-3 also consider qSOFA as a prompt to identify possible infection [1].
In many patients admitted to the ED with sepsis the severity of their illness is not directly clear. The presence of a life-threatening infection can easily be overlooked. The use of screening tools in the ED can aid in early recognition of patients with sepsis, resulting in early initiation of effective and complete treatment. This requires screening tools with a high sensitivity. SIRS has been criticized for being too sensitive, while lacking specificity in recognizing sepsis, and it is therefore not an ideal screening tool. As qSOFA performed better than SIRS in hospitalized patients, it has been proposed that qSOFA is preferred to SIRS. Alternatively, early warning scores, such as the National Early Warning Score (NEWS), are already recommended for use in the ED, and should therefore also be considered [11]. NEWS was introduced in 2012 by the Royal College of Physicians, who aimed to provide a standardised early warning score. This score is used for early detection of patients at risk for deterioration but is not specific for sepsis. NEWS comprises of seven parameters (i.e. respiratory rate, oxygen saturation, supplemental oxygen, body temperature, systolic blood pressure, heart rate, AVPU score) with a maximum of twenty points. In clinical practice cut-off values of 1–4, 5–6 and ≥7, respectively for low, medium and high risk are used. NEWS was primarily developed for use on the wards, however NEWS was also tested for use in the ED and in the prehospital setting [12, 13]. For use in the ED a cut-off value of ≥7 is suggested.
The aim of this study was to determine the prognostic value of qSOFA in predicting mortality in comparison to SIRS and NEWS in patients presenting to the ED with suspected sepsis.
Methods
Study design and setting
This was a retrospective cohort study nested in a large anonymous database of patients visiting the ED of the Erasmus University Medical Center, Rotterdam, the Netherlands (Erasmus MC), which is the largest tertiary referral center in The Netherlands. The ED is an open access department with approximately 30,000 annual visits. Patients are strongly encouraged to see a general practitioner before visiting the ED. The database of the ED consists of all patients presenting to the ED. This database holds information of patients from January 2012 and onwards, on both clinical and vital parameters, laboratory results, other diagnostic procedures and treatments. The data was extracted from the electronic health records every two weeks through May 2017. Random samples were manually checked for concordance.
Selection of participants
In our consecutive cohort, we included patients with suspected sepsis visiting the ED between June 1st 2012 and May 31st 2016. Suspected sepsis was defined as either the initiation of non-prophylactic intravenous antibiotic therapy during their ED visit or the collection of any culture (i.e. blood cultures, urine cultures, wound cultures, throat swabs, sputum cultures and cultures of cerebrospinal fluid) or viral diagnostics (i.e. polymerase chain reaction (PCR) on blood and stool samples, on throat swabs and on cerebrospinal fluids) during the index visit. Rapid diagnostic testing for viral or bacterial infections was not possible during the study period. Patients who presented with symptoms directly related to trauma were excluded. A comprehensive search in the database identified all patients who met this definition.
Measurements and outcomes
Demographic data (i.e. age, sex), vital parameters (i.e. blood pressure, body temperature, respiratory rate, peripheral oxygen saturation, consciousness level according to AVPU scale or GCS), laboratory testing performed, acuity level according to Manchester Triage System (MTS) category, and supplemental oxygen therapy were derived from the database.
The AVPU scale is a system to score the mental status and is an acronym of ‘Alert, Verbal, Pain, Unresponsive’ [14]. When AVPU was not scored, GCS was used, and vice versa. Only the first vital parameters were retrieved as the aim of the study was to assess the ability of the different prompts to screen for short-term mortality at ED presentation. White blood cell count was retrieved for all patients when available. Data on all-cause mortality was obtained from patient records and 10- and 30-day mortality was calculated. Mortality data was retrieved from the patient records, which are linked to municipal mortality data. Subsequently, we assessed whether mortality was directly sepsis-related or not.
We calculated qSOFA, SIRS and NEWS and formed groups using cut-off values most indicative for poor outcome (qSOFA≥2, SIRS≥2, and NEWS≥7)(Table 1) [2, 4, 11]. The Medical Ethics Committee of the Erasmus MC reviewed the study and deemed exempt.
Statistical analysis
Data was summarized using mean, median, interquartile range (IQR) and standard deviation (SD) when appropriate. Missing or clinically implausible data was replaced by multiple imputation. This method is valid even when large sets of data are missing[15]. Missing values within the parameters were imputed five times using non-missing parameters. Furthermore, imputation was based on a distribution of the observed data to preclude that implausible values would replace the missing value. After imputation, five complete datasets were available. In each dataset the SIRS, qSOFA and NEWS scores were recalculated using the imputed variables. Whenever possible, results were pooled. When pooling was not possible, single imputation was used. The primary outcome was all-cause mortality within 10- and 30-days after ED presentation.
Patient characteristics were compared using the two-sampled t-test, Mann-Whitney U test, and chi-squared test based on the distribution of the data. Univariate regression analysis was used for association between the different parameters and 10- and 30-day mortality to determine which variable is the best predictor. This predictor is characterized by the largest LRχ2 and a high explained variance (i.e. R2 close to one).
Logistic regression was used to obtain the odds for 10- and 30-day mortality based on individual scores. The predictive performances of qSOFA, SIRS, and NEWS were expressed as discrimination (area under the Receiver Operating Characteristic-curve) and calibration. Calibration represents how mortality predictions resemble the observed mortality, which was measured by the Hosmer-Lemeshow goodness-of-fit test and expressed as a χ2-value and accessory p-value. Subsequently, sensitivity, specificity and positive- and negative predictive values were calculated for the different cut-off points. The Youden’s J statistic was calculated to assess the optimal cut-off point for the different scores. A p-value <0.05 was considered statistically significant. Analyses were undertaken using Statistical Package for the Social Science (SPSS) version 21 and R statistics version 3.1.3. (2015-03-09).
Results
Patient characteristics
A total of 120,177 ED visits in 75,428 unique patients were recorded between June 1st 2012 and May 31st 2016. 21,326 patient records were excluded as their ED visits were related to trauma, leaving 54,102 patients for analysis. 3,351 patients received intravenous antibiotic therapy in the ED. Bacterial cultures and viral diagnostics were collected from 7,302 patients during their ED visit. In total, 8,204 patients were analyzed (Fig 1). The majority of patients were male (55.9%), and the median age was 57.0 (IQR 41.0–67.0). In total, 74.6% of patients were hospitalized (Table 2). 10-day and 30-day mortality was 3.5% (286) and 6.0% (490), respectively. Of the 490 deceased patients, 64,7% died in the hospital. Patients who died were significantly older, and had higher heart rates, lower systolic blood pressures, lower oxygen saturation and higher respiratory rates during ED presentation. 18,4% of the deceased patients had positive cultures. The cause of death could be retrieved from the patient records in all 490 deceased patients. In 63.4% of patients their death was directly related to sepsis.
Flowchart of patients who met inclusion/exclusion criteria.
Performance of the models
Univariate regression analysis showed that oxygen therapy during ED presentation—a variable within NEWS—was the best predictor for mortality (LRχ2 = 335.73), although the explained variation was low (r2 = 0.110). Other strong predictors included systolic blood pressure and mental status (Table 3).
NEWS performed substantially better than qSOFA and SIRS in predicting both 10-day mortality (AUC [95% CI]: 0.837 [0.812, 0.861], 0.744 [0.708, 0.78] and 0.646 [0.613, 0.679] respectively) and 30-day mortality (0.779 [0.755, 0.804], 0.697 [0.667, 0.726] and 0.631 [0.605, 0.656] respectively) (Figs 2 and 3).
Calibration for NEWS showed a χ2 = 10.743 and p-value = 0.217, compared to χ2 = 6.915 and p-value = 0.032 for qSOFA, and χ2 = 22.827 and p-value = 0.004 for SIRS. The non-significant p-value indicates that the mortality rates between the observed and the predicted values were statistically equivalent.
qSOFA showed the highest specificity, followed by NEWS and SIRS. Sensitivity was highest in SIRS, followed by NEWS and qSOFA. Using Youden’s J statistic, the optimal cut-off points for 10-day mortality were qSOFA ≥1, SIRS≥2 and NEWS≥7, and for 30-day qSOFA ≥1, SIRS≥3 and NEWS≥7 (Table 4).
Discussion
In this retrospective observational study of patients visiting the ED with a suspected sepsis we found that NEWS was superior to qSOFA and SIRS in predicting 10- and 30-day mortality for both discrimination and calibration. The different prompts all have different sensitivities and specificities for mortality. qSOFA has the highest specificity and lowest sensitivity, SIRS has the lowest specificity and highest sensitivity. NEWS has both an intermediate sensitivity and specificity, but is the best overall predictor in distinguishing high risk from low risk patients. NEWS has a lower sensitivity resulting in a significant number of false negatives, i.e. not all the patients who eventually died were identified with NEWS. NEWS was the only model with a good agreement between the expected and observed outcomes, i.e. calibration. However, none of the prediction models succeeded to fulfil all performance assessments, which would ideally be the case. Subsequent measurements of NEWS (e.g. hourly) will potentially identify patients who deteriorate during the stay in the ED and may improve sensitivity. We conclude that at presentation to the ED NEWS can be used as an alternative screening tool for patients with suspected sepsis who are at risk for deterioration, multi-organ failure, and subsequently death.
Our findings support the increasing data that suggests that the NEWS score is a useful screening tool in the ED, although its use has not fully been validated in the ED setting. Jo et al. studied the NEWS combined with serum lactate in predicting mortality in the general adult ED population and found an excellent discrimination (AUC = 0.96) for predicting two-day mortality [16]. The NEWS score as measured in the prehospital setting showed good correlation (p<0.001) with hospital disposition [17]. Our study confirms the findings by Churpek et al. which support the introduction of the NEWS score in the ED. However, they studied patients outside the ICU and not only ED patients. And they primarily measured the performance of the different prompts based on the worst vital signs. NEWS had the highest performance in predicting in-hospital mortality in ED patients compared to qSOFA and SIRS (AUC = 0.77, AUC = 0.69 and AUC = 0.65 respectively). We used vital parameters at presentation in the ED and found similar results. In the Churpek et al. study a NEWS threshold of ≥7 is suggested. This threshold is also recommended by the Royal College of Physicians[11]. We were able to confirm this threshold using our data. In a cohort study by Sbiti-Rohr et al. in patients with community-acquired pneumonia, the NEWS score in the ED was significantly higher for those who died within 30 days after presentation than for survivors [18]. These results are similar to a study of patients presenting to the ED with acute dyspnea; survivors had significantly lower NEWS scores at ED presentation [19].
The NEWS was also studied in patients suspected of sepsis. Corfield et al. found that an increased NEWS on arrival at the ED was associated with mortality in patients who met the sepsis criteria as defined by Bone et al. (odds ratio 1.95 to 5.64) [20].
Most prediction scores include measurements which are subject to interpretation. A study on the interrater agreement of GCS assessed at the ED yielded low agreement [21]. Semler et al. showed that in hospitalized patients recorded respiratory rates were higher than directly observed measurements. Also, the recorded rates were more likely to be 18 or 20 breaths/minute [22]. We expect that parameters that are not acquired automatically are subject to confounding by disease severity and were more likely to be measured and noted when one would expect a deviant result [23, 24]. Therefore, for the proper use of the NEWS, qSOFA and SIRS these measurements should be routinely performed in a structural way.
Specific scoring systems are used as an alternative to the NEWS to predict sepsis-related mortality in ED patients. The SIRS criteria, as introduced by Bone in 1992, were studied as a prediction tool for mortality and most studies show that an increase in SIRS items reflects an increased risk of mortality, ranging from 1.4% to 12% when no SIRS criteria were met and increasing to approximately 36% for four SIRS items [25, 26]. In Sepsis-3, the qSOFA was introduced as a simple tool to detect deterioration and predict mortality in departments outside the ICU. Simultaneously, SIRS criteria were abandoned from the new sepsis definition after criticism of its low specificity. The qSOFA≥2 resembles a three to 14-fold increase in mortality risk[4].
qSOFA has been challenged as a prompt in the ED to identify patients with an increased risk for sepsis-related mortality ever since its introduction. Despite a high specificity (84–96%), the qSOFA has low sensitivity (13–53%) [8, 27]. This low sensitivity can be explained by the fact that the qSOFA is composed of vital parameters representing late symptoms of deterioration (e.g. altered mental status due to inadequate perfusion of the brain) [28, 29]. In addition, qSOFA was derived in a cohort of critically ill patients, in which 11% of the patients were admitted in the ICU [4]. These patients represent a selected population compared to all patients who visit the ED, therefore, selection bias may be present. Furthermore, qSOFA was developed on the most aberrant results in serial vital parameter measurements. This approach may ameliorate the ability to predict mortality, but it restricts the utility as a prompt for early identification of patients at risk directly at ED presentation. All these arguments mainly affect the sensitivity and can influence the predictive performance of qSOFA. To increase sensitivity, Park et al. proposed the use of the qSOFA cut-off point of ≥1 instead of 2 for patients in the ED, resulting in an increase in sensitivity from 53.0% to 82.0%. This is in line with our findings. Changing the cut-off to 1 would increase the usability of qSOFA as a screening tool at cost of specificity. However, NEWS still has a higher sensitivity and a better predictive performance.
Strengths and limitations
This study has a number of strengths and limitations. The major strength of our study is that we used a large consecutive dataset with many relevant parameters directly derived from electronic patient records with mortality data directly acquired from municipality data.
Our study also has several limitations. The first limitation of this study is its retrospective design using data from a single tertiary care center. In our center we treat many patients with congenital and acquired immunodeficiencies (e.g. patients with organ or bone marrow transplantation, chemotherapy), which may limit the generalizability. The database contained missing values, which were replaced by multiple imputation. Multiple imputation has also been used in other sepsis-related studies [4, 8, 30]. Respiratory rate was most frequently missing and, as mentioned earlier, availability of respiratory rate might be an indicator of confounding by indication, as it is more often measured in patients who are deemed more critically ill[23]. A second limitation is the definition of the study population. As there is no gold standard for defining an infection, the study population was difficult to determine. We based our inclusion criteria on the definition of Seymour et al. [4] but modified the criteria to incorporate the largest group of patients who were suspected for infection and at risk for sepsis. Both microbial diagnostics and initiation of antibiotics were used as a proxy for a clinically suspected sepsis. These inclusion criteria could possibly bias against people with viral disease, as no antibiotics given and cultures are not routinely performed. However, in the most critically ill patients cultures are taken and antibiotics are started empirically in clinical practice, regardless of the suspected pathogen (e.g. virus, bacteria). Furthermore, we also included viral cultures such as throat swabs and stool cultures, but these were a minority as compared to blood cultures (289 and 46 vs. 6552). Therefore, the chance of bias due viral sepsis is limited.
Last, to determine the best screening tool at presentation in the ED, we chose to use only the first recorded vital signs for calculation of NEWS, qSOFA and SIRS. We are aware that rapid changes in vital parameters could be indicative for a higher risk for mortality and that people may deteriorate during their ED visit. However, the duration of ED stay is intended to be very limited. Choosing to only use the first vital parameters may limit the predictive ability of the different models. However, in clinical practice the first vital parameters are used to determine the severity of the patient’s condition and, therefore, to triage patients in urgent and non-urgent. Using first available parameters in this study actually reflects clinical practice and in our opinion is a valid method to test predictive performance upon ED presentation, with results comparable to using the worst vital parameters[31].
Conclusions
In conclusion, the NEWS is more accurate in predicting 10- and 30-day mortality than qSOFA and SIRS in patients suspected of sepsis on initial presentation to the ED. Our finding suggests that the introduction of the NEWS in the ED with subsequent measurements should be further studied. This will potentially aid the early detection of all patients at risk for deterioration in the ED including those at risk of sepsis-related mortality.
Supporting information
S1 Table. Sensitivity (95% CI), specificity (95% CI), positive predictive value, negative predictive value and Youden’s index for different cut-off values for 10- and 30-day mortality.
║ are the predefined cut-off values which are most indicative for a poor outcome. ¶ representing the optimal cut-off points. Abbreviations: CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value; SIRS, systemic inflammatory response syndrome; qSOFA, quick sepsis-related organ failure assessment; NEWS, national early warning score.
https://doi.org/10.1371/journal.pone.0211133.s001
(DOCX)
References
- 1. Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). Jama-J Am Med Assoc. 2016;315(8):801–10. WOS:000370700500015. pmid:26903338
- 2. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992;101(6):1644–55. Epub 1992/06/01. S0012-3692(16)38415-X [pii]. pmid:1303622.
- 3. Levy MM, Fink MP, Marshall JC, Abraham E, Angus D, Cook D, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Crit Care Med. 2003;31(4):1250–6. Epub 2003/04/12. pmid:12682500.
- 4. Seymour CW, Liu VX, Iwashyna TJ. Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) (vol 315, pg 762, 2016). Jama-J Am Med Assoc. 2016;315(20):2237–. WOS:000376343000031.
- 5. Simpson SQ. New Sepsis Criteria: A Change We Should Not Make. Chest. 2016;149(5):1117–8. Epub 2016/03/02. S0012-3692(16)41523-0 [pii]. pmid:26927525.
- 6. Carneiro AH, Povoa P, Gomes JA. Dear Sepsis-3, we are sorry to say that we don't like you. Rev Bras Ter Intensiva. 2017;29(1):4–8. Epub 2017/04/27. S0103-507X2017000100004 [pii] pmid:28444066; PubMed Central PMCID: PMC5385979.
- 7. Goulden R, Hoyle MC, Monis J, Railton D, Riley V, Martin P, et al. qSOFA, SIRS and NEWS for predicting inhospital mortality and ICU admission in emergency admissions treated as sepsis. Emerg Med J. 2018;35(6):345–9. Epub 2018/02/23. emermed-2017-207120 [pii] pmid:29467173.
- 8. Askim A, Moser F, Gustad LT, Stene H, Gundersen M, Asvold BO, et al. Poor performance of quick-SOFA (qSOFA) score in predicting severe sepsis and mortality—a prospective study of patients admitted with infection to the emergency department. Scand J Trauma Resusc Emerg Med. 2017;25(1):56. Epub 2017/06/11. [pii]. pmid:28599661; PubMed Central PMCID: PMC5466747.
- 9. Canet E, Taylor DM, Khor R, Krishnan V, Bellomo R. qSOFA as predictor of mortality and prolonged ICU admission in Emergency Department patients with suspected infection. J Crit Care. 2018;48:118–23. Epub 2018/09/04. S0883-9441(18)30768-8 [pii] pmid:30176527.
- 10. Hwang SY, Jo IJ, Lee SU, Lee TR, Yoon H, Cha WC, et al. Low Accuracy of Positive qSOFA Criteria for Predicting 28-Day Mortality in Critically Ill Septic Patients During the Early Period After Emergency Department Presentation. Ann Emerg Med. 2018;71(1):1–9 e2. Epub 2017/07/04. S0196-0644(17)30658-3 [pii] pmid:28669551.
- 11.
Royal College of Physicians. National Early Warning Score (NEWS): Standardising the assessment of acute illness severity in the NHS. Report of a working party. London: RCP; 2012.
- 12. Kivipuro M, Tirkkonen J, Kontula T, Solin J, Kalliomaki J, Pauniaho SL, et al. National early warning score (NEWS) in a Finnish multidisciplinary emergency department and direct vs. late admission to intensive care. Resuscitation. 2018;128:164–9. Epub 2018/05/19. S0300-9572(18)30231-4 [pii] pmid:29775642.
- 13. Hoikka M, Silfvast T, Ala-Kokko TI. Does the prehospital National Early Warning Score predict the short-term mortality of unselected emergency patients? Scand J Trauma Resusc Emerg Med. 2018;26(1):48. Epub 2018/06/09. [pii]. pmid:29880018; PubMed Central PMCID: PMC5992854.
- 14.
Alexander RH, Proctor HJ, American College of Surgeons, Committee on Trauma. Advanced Trauma Life Support Program For Physicians: ATLS. 5th ed. Chicago, IL: American College of Surgeons; 1993.
- 15. Pedersen AB, Mikkelsen EM, Cronin-Fenton D, Kristensen NR, Pham TM, Pedersen L, et al. Missing data and multiple imputation in clinical epidemiological research. Clin Epidemiol. 2017;9:157–66. Epub 2017/03/30. [pii]. pmid:28352203; PubMed Central PMCID: PMC5358992.
- 16. Jo S, Yoon J, Lee JB, Jin Y, Jeong T, Park B. Predictive value of the National Early Warning Score-Lactate for mortality and the need for critical care among general emergency department patients. J Crit Care. 2016;36:60–8. Epub 2016/11/05. S0883-9441(16)30138-1 [pii] pmid:27546749.
- 17. Shaw J, Fothergill RT, Clark S, Moore F. Can the prehospital National Early Warning Score identify patients most at risk from subsequent deterioration? Emerg Med J. 2017. Epub 2017/05/16. emermed-2016-206115 [pii] pmid:28501815.
- 18. Sbiti-Rohr D, Kutz A, Christ-Crain M, Thomann R, Zimmerli W, Hoess C, et al. The National Early Warning Score (NEWS) for outcome prediction in emergency department patients with community-acquired pneumonia: results from a 6-year prospective cohort study. BMJ Open. 2016;6(9):e011021. Epub 2016/09/30. bmjopen-2015-011021 [pii] pmid:27683509; PubMed Central PMCID: PMC5051330.
- 19. Bilben B, Grandal L, Sovik S. National Early Warning Score (NEWS) as an emergency department predictor of disease severity and 90-day survival in the acutely dyspneic patient—a prospective observational study. Scand J Trauma Resusc Emerg Med. 2016;24:80. Epub 2016/06/03. [pii]. pmid:27250249; PubMed Central PMCID: PMC4890514.
- 20. Corfield AR, Lees F, Zealley I, Houston G, Dickie S, Ward K, et al. Utility of a single early warning score in patients with sepsis in the emergency department. Emerg Med J. 2014;31(6):482–7. Epub 2013/03/12. emermed-2012-202186 [pii] pmid:23475607.
- 21. Gill MR, Reiley DG, Green SM. Interrater reliability of Glasgow Coma Scale scores in the emergency department. Ann Emerg Med. 2004;43(2):215–23. Epub 2004/01/30. S019606440300814X [pii]. pmid:14747811.
- 22. Semler MW, Stover DG, Copland AP, Hong G, Johnson MJ, Kriss MS, et al. Flash mob research: a single-day, multicenter, resident-directed study of respiratory rate. Chest. 2013;143(6):1740–4. Epub 2012/12/01. S0012-3692(13)60406-7 [pii] pmid:23197319; PubMed Central PMCID: PMC3747725.
- 23. Sjoding MW, Luo K, Miller MA, Iwashyna TJ. When do confounding by indication and inadequate risk adjustment bias critical care studies? A simulation study. Crit Care. 2015;19:195. Epub 2015/05/01. [pii]. pmid:25925165; PubMed Central PMCID: PMC4432515.
- 24. Cretikos MA, Bellomo R, Hillman K, Chen J, Finfer S, Flabouris A. Respiratory rate: the neglected vital sign. Med J Aust. 2008;188(11):657–9. Epub 2008/06/03. cre11027_fm [pii]. pmid:18513176.
- 25. Asayama K, Aikawa N. Evaluation of systemic inflammatory response syndrome criteria as a predictor of mortality in emergency patients transported by ambulance. Keio Journal of Medicine. 1998;47(1):19–27. pmid:9560528.
- 26. Jones GR, Lowes JA. The systemic inflammatory response syndrome as a predictor of bacteraemia and outcome from sepsis. Qjm. 1996;89(7):515–22. pmid:8759492.
- 27. Park HK, Kim WY, Kim MC, Jung W, Ko BS. Quick sequential organ failure assessment compared to systemic inflammatory response syndrome for predicting sepsis in emergency department. J Crit Care. 2017;42:12–7. Epub 2017/06/26. S0883-9441(17)30395-7 [pii] pmid:28647650.
- 28. Simpson SQ, City K. New Sepsis Criteria A Change We Should Not Make. Chest. 2016;149(5):1117–8. WOS:000375487600007. pmid:26927525
- 29. Sprung CL, Schein RMH, Balk RA. The new sepsis consensus definitions: the good, the bad and the ugly. Intens Care Med. 2016;42(12):2024–6. WOS:000387846500024. pmid:27815588
- 30. Raith EP, Udy AA, Bailey M, McGloughlin S, MacIsaac C, Bellomo R, et al. Prognostic Accuracy of the SOFA Score, SIRS Criteria, and qSOFA Score for In-Hospital Mortality Among Adults With Suspected Infection Admitted to the Intensive Care Unit. JAMA. 2017;317(3):290–300. Epub 2017/01/24. 2598267 [pii] pmid:28114553.
- 31. Churpek MM, Snyder A, Han X, Sokol S, Pettit N, Howell MD, et al. Quick Sepsis-related Organ Failure Assessment, Systemic Inflammatory Response Syndrome, and Early Warning Scores for Detecting Clinical Deterioration in Infected Patients outside the Intensive Care Unit. Am J Respir Crit Care Med. 2017;195(7):906–11. Epub 2016/09/21. pmid:27649072; PubMed Central PMCID: PMC5387705.