Background The third international consensus definition for sepsis recommended use of a new prognostic tool, the quick Sequential Organ Failure Assessment (qSOFA), based on its ability to predict inhospital mortality and prolonged intensive care unit (ICU) stay in patients with suspected infection. While several studies have compared the prognostic accuracy of qSOFA to the Systemic Inflammatory Response Syndrome (SIRS) criteria in suspected sepsis, few have compared qSOFA and SIRS to the widely used National Early Warning Score (NEWS).
Methods This was a retrospective cohort study carried out in a UK tertiary centre. The study population comprised emergency admissions in whom sepsis was suspected and treated. The accuracy for predicting inhospital mortality and ICU admission was calculated and compared for qSOFA, SIRS and NEWS.
Results Among 1818 patients, 53 were admitted to ICU (3%) and 265 died in hospital (15%). For predicting inhospital mortality, the area under the receiver operating characteristics curve for NEWS (0.65, 95% CI 0.61 to 0.68) was similar to qSOFA (0.62, 95% CI 0.59 to 0.66) (test for difference, P=0.18) and superior to SIRS (P<0.001), which was not predictive. The sensitivity of NEWS≥5 (74%, 95% CI 68% to 79%) was similar to SIRS≥2 (80%, 95% CI 74% to 84%) and higher than qSOFA≥2 (37%, 95% CI 31% to 43%). The specificity of NEWS≥5 (43%, 95% CI 41% to 46%) was higher than SIRS≥2 (21%, 95% CI 19% to 23%) and lower than qSOFA≥2 (79%, 95% CI 77% to 81%). The negative predictive value was 88% (86%–90%) for qSOFA, 86% (82%–89%) for SIRS and 91% (88%–93%) for NEWS. Results were similar for the secondary outcome of ICU admission.
Conclusion NEWS has equivalent or superior value for most test characteristics relative to SIRS and qSOFA, calling into question the rationale of adopting qSOFA in institutions where NEWS is already in use.
- clinical assessment
- infectious diseases
Statistics from Altmetric.com
What is already known on this subject
Multiple studies have demonstrated that the quick Sequential Organ Failure Assessment (qSOFA) score has higher specificity but lower sensitivity than the Systemic Inflammatory Response Syndrome (SIRS) criteria for predicting adverse outcomes in sepsis.
Only one previous study has compared these scores to the National Early Warning Score (NEWS).
What this study adds
In this retrospective cohort of patients in whom sepsis was suspected and treated, NEWS had similar or superior values across most measures of prognostic accuracy compared with qSOFA or SIRS.
This study calls into question the value of qSOFA in institutions where NEWS is already in use.
Sepsis is a significant cause of morbidity and mortality. There are an estimated 150 000 sepsis admissions in the UK annually, with a 30% mortality rate resulting in over 44 000 deaths.1 2 Most patients with sepsis are admitted to hospital via the ED. Early recognition and management through use of care bundles is associated with lower mortality.3
A key strategy for improving sepsis management is to identify the subgroup of patients with infection who are at high risk of adverse outcomes. This may help clinicians speed up treatment such as administration of antibiotics. The third international consensus definition (Sepsis 3) recommended use of a new tool to identify patients at high risk of death and prolonged intensive care unit (ICU) stay among those with suspected infection, the quick Sequential Organ Failure Assessment (qSOFA) score.4 Sepsis was previously identified and defined using the Systemic Inflammatory Response Syndrome (SIRS) criteria, though this definition was limited by its poor specificity.5 6 Meanwhile, the National Early Warning Score (NEWS) has been widely adopted in the UK NHS as a tool to assess and monitor the clinical condition of hospital patients.7 Despite the differing initial aims of these three scoring systems, all may be used by clinicians to evaluate illness severity and prognosis in patients with suspected infection or sepsis. It is thus important to determine their relative accuracy in achieving this goal.
Following the publication of Sepsis 3, several studies have sought to externally validate the qSOFA score by assessing its prognostic accuracy in comparison to SIRS criteria. In general, qSOFA appears to have a higher specificity but lower sensitivity than SIRS criteria for predicting mortality and ICU admission. It was also shown to have a slightly higher area under the receiver operating characteristics curve (AUROC).8–12 Only one prior study—carried out in a single US hospital—has also compared these scores with NEWS.8 In this study, NEWS retained the sensitivity of SIRS but had higher specificity as well as a higher AUROC than both qSOFA and SIRS. If these findings can be replicated in a different population, this may have important implications for clinical practice across the UK and internationally. Many institutions are now actively considering the adoption of qSOFA to replace or supplement SIRS criteria. If NEWS, which is already routinely used in many hospitals, can be shown to be of equivalent or higher prognostic accuracy, the rationale for using either qSOFA or SIRS is called into question.
This study seeks to evaluate the prognostic accuracy of qSOFA, SIRS and NEWS for predicting inhospital mortality and ICU admission in emergency admissions prospectively identified by the clinical team as having suspected sepsis and treated as such.
This was a retrospective cohort study of all adult patients presenting to the ED or medical admissions unit (MAU) of the Royal Liverpool University Hospital (RLUH) between April 2016 and May 2017, in whom sepsis was suspected and treated by the clinical team. The RLUH is an 850-bed tertiary centre with around 90 000 ED attendances annually. For patients with multiple attendances, one attendance was randomly selected for study inclusion.
Patients in whom sepsis is suspected at RLUH are referred to the sepsis specialist nurses. Referrals can be made by doctors or nurses and are typically made after initial triage or clinical assessment in ED or MAU. The sepsis nurses provide 7-day onsite daytime coverage as well as reviewing overnight ED attendance records each morning for any additional cases. Following patient review in the ED or MAU, if the clinical team and sepsis nurses agree that sepsis is sufficiently likely to be treated as such, the sepsis nurses complete a standardised form in the electronic medical record (EMR), which includes the qSOFA, SIRS and NEWS scores on arrival as well as the suspected source of infection and other aspects of patient care. Our study population thus comprised all patients who had a sepsis form completed in their EMR during the study period, based on a referral made from the ED or MAU. These data are collected for local and national audit and quality improvement initiatives. The study period comprised the maximal time window for which data were available at the time of conducting this study. The EMR system allows for the automated extraction of data from all patient records with a completed sepsis form, along with demographic details and clinical outcomes. Any patients for whom qSOFA, SIRS or NEWS scores were not recorded in the electronic form, had their paper records manually reviewed by a single doctor or sepsis nurse to retrieve missing values. Incomplete values were expected to be highest for qSOFA, as routine electronic recording of the new score was only gradually adopted over the course of the study period.
The primary outcome was inhospital mortality. Secondary outcomes were ICU admission and a composite of inhospital mortality or ICU admission, recorded as positive if either outcome occurred.
SIRS criteria are defined as a heart rate >90 beats per minute, a respiratory rate >20 breaths per minute, a temperature <36°C or >38°C and a white blood cell count <4000/mm3 or >12 000/mm3. A positive score is defined as ≥2 out of 4.13 qSOFA criteria are a systolic blood pressure ≤100 mm Hg, a respiratory rate ≥ 22 breaths per minute and a Glasgow Coma Scale score<15. A positive score is defined as ≥2 out of 3.4 The NEWS score ranges from 0 to 20 and is based on respiratory rate, oxygen saturations, use of supplemental oxygen, temperature, systolic blood pressure, pulse rate and level of consciousness. A positive score is defined as ≥5 out of 20, the threshold suggested as representing a ‘red score’ indicative of significant physiological derangement.7
The sensitivity, specificity, AUROC, positive predictive value (PPV), negative predictive value (NPV) and positive and negative likelihood ratio for predicting inhospital mortality and ICU admission were calculated for each scoring system. In the primary analysis, the predefined thresholds of a positive score for each scoring system were used. Exploratory analyses then determined the optimal cutpoint for each scoring system, based on the threshold with the highest Youden index. The sensitivity and specificity of the different scoring systems for the primary outcome were compared using McNemar’s test, and the AUROC compared using DeLong’s method.14 Starting with the conventional significant threshold of P=0.05, applying the Bonferroni correction for multiple testing (0.05/9) gave a significance threshold of 0.006. Analyses were carried out in Stata V.15.0 (Stata, College Station, Texas, USA), using the commands roctab, roccomp, diagt and cutpt.15 16
For those still missing values of the scoring systems following automatic and manual data extraction, we evaluated if missingness was predicted by other variables—age, sex, inhospital death, ICU admission, value of non-missing scores—using logistic regression. If there was evidence that data were missing at random, multiple imputation would be used, otherwise a complete case analysis would be performed.
The study data comprised data recorded in the medical record as part of routine patient care. In line with standard practice, patient consent for the retrospective analysis of these data was not deemed necessary.
There were 2158 attendances to the ED or MAU during the study period in which sepsis was suspected and treated by the clinical team. This included 1942 unique patients. In the automated data extraction, 1117 patients (58%) had missing qSOFA scores, 103 (5%) missing SIRS scores and 335 (17%) missing NEWS scores. After manual review of paper charts for missing scores, the value of all three scoring systems on arrival was available for 1818 of these patients (94%), forming our study population. We found no evidence that missingness was predicted by other variables; thus, a complete case analysis was performed. The mean age of patients was 68 years and 49% were female. Fifty-three patients were admitted to ICU (3%) and 265 died in hospital (15%). Patient characteristics are summarised in table 1.
For predicting inhospital mortality among all patients with suspected sepsis, qSOFA was the most specific (79%, 95% CI 77% to 81%) but least sensitive (37%, 95% CI 31% to 43%) scoring system. SIRS was the most sensitive (80%, 95% CI 74% to 84%) but least specific (21%, 95% CI 19% to 23%). NEWS had an intermediate sensitivity (74%, 95% CI 68% to 79%) and specificity (43%, 95% CI 41% to 46%) (table 2). The sensitivity of SIRS was statistically significantly higher than qSOFA (P<0.001), but not significantly higher than NEWS when using the Bonferroni correction (P=0.02). The sensitivity of NEWS was statistically significantly higher than qSOFA (P<0.001). The specificity of qSOFA was statistically significantly higher than NEWS (P<0.001) and SIRS (P<0.001) and that of NEWS statistically significantly higher than SIRS (P<0.001).
NEWS had a similar AUROC (0.65, 95% CI 0.61 to 0.68) to qSOFA (0.62, 95% CI 0.59 to 0.66), with no statistically significant difference (P=0.18) (figure 1 and table 2). Relative to SIRS (AUROC 0.49, 95% CI 0.45 to 0.52), the AUROC was statistically significantly higher for NEWS (P<0.001) and qSOFA (P<0.001). Results were similar for ICU admission (figure 2 and table 2) as well as the composite of inhospital death or ICU admission (online supplementary table 1). The PPV and NPV of all three scoring systems were similar. The likelihood ratio for qSOFA (1.78, 95% CI 1.48 to 2.14) was higher than NEWS (1.30, 95% CI 1.19 to 1.41) and SIRS (1.01, 95% CI 0.94 to 1.08).
Exploratory analyses using the Youden index suggested a qSOFA of ≥1 (sensitivity 77%, specificity 41%) or NEWS of ≥7 (sensitivity 56%, specificity 67%) may be optimal thresholds for predicting inhospital mortality (online supplementary tables 2–4).
Our findings suggest that no scoring system has both high sensitivity and specificity for predicting adverse outcomes in sepsis on hospital arrival. However, in terms of AUROC, NEWS is equivalent to qSOFA and more accurate than SIRS, while avoiding the extremely low sensitivity of qSOFA, although at the cost of lower specificity. These results are broadly consistent with the one previous study that compared all three scoring systems,8 though our study had lower sensitivity and higher specificity for all scoring systems. Our results are also consistent with other studies showing a problematically low specificity for SIRS9 11 and low sensitivity for qSOFA10 12 17 18 for predicting adverse outcomes. A further non-comparative study of NEWS in a UK ED population had similar findings to ours.19 The fact that NEWS incorporates a greater number of physiological parameters than qSOFA and SIRS, including most of their component parts, likely accounts for its favourable prognostic accuracy. It could be argued that any scoring system for identifying sepsis should lean towards higher sensitivity rather than specificity, as the costs of false-negatives (delayed or missed treatment) are arguably much greater than those of false positives (unnecessary antibiotics). However, the AUROC for SIRS was so low as to suggest it has very limited clinical utility.
The exploratory analysis of optimal cutpoints suggested a qSOFA score of 1 had a relatively favourable prognostic value. This supports the recommendation made by the Sepsis 3 taskforce to use respiratory rate, blood pressure and mental status as a set of parameters which capture much of the prognostic power of more detailed scoring systems. However, given that all the components of NEWS are routinely collected in most hospitals as part of basic triage and nursing care, the value of calculating a score based on a subset of these measure is not clear.
The poor performance of all thee scoring systems in our study highlights the complexity of predicting outcomes in suspected sepsis, especially in its early stages. While SIRS and qSOFA were developed specifically to identify or prognosticate in sepsis, neither appear to have both sufficient sensitivity and specificity. Indeed neither are truly sepsis-specific scoring systems, having similar prognostic characteristics in patients without infection as in those with infection.5 17 Other scoring systems, such as PIRO and MEDS, have been shown to have better predictive capacity than SIRS,20 21 but are not in widespread use. Clinicians who work daily to identify sepsis are subject to several, sometimes conflicting, guidelines. In the UK, the NICE quality standards recommend NEWS22 and the NICE guidelines further specify various ‘risk criteria’ to guide treatment,23 while the international consensus definition recommends qSOFA,4 and US national quality standards are based around SIRS.24 The limitations of all scoring systems in our study highlight the fact that they should be used as only one part of a much broader clinical assessment and that caution must be exercised in building sepsis identification and alert systems reliant solely on these criteria.
We lacked data on patient comorbidities and cause of death, limiting our ability to determine the specific role of sepsis in patient outcomes and the extent to which these would be altered by sepsis-specific interventions. While the goal of this study is only to evaluate prognostic scoring systems in sepsis, such systems are only useful in so far as they identify patients in whom early interventions would be appropriate and effective. The patients who died in our cohort were significantly older than the overall study population (mean age 78 years versus 68 years), and the vast majority of them were not admitted to ICU (94%). This suggests that invasive organ support, a key component of sepsis treatment for patients unresponsive to initial therapy, was not deemed necessary or appropriate for most of these patients. This would be consistent with sepsis being a terminal event in many non-infectious chronic diseases and rarely the underlying cause of death itself.25
Another limitation relates to the time at which prognostic scores were calculated. Improved accuracy may have been achieved had we taken the highest value of each scoring system throughout a patient’s stay in the ED or MAU, as some other studies have done.8 9 11 However, it can be argued that the initial score more accurately represents the basis on which most clinical decisions are made in this ED and MAU.
There may be a risk of bias in our study from missing data. While we could determine that missingness was not related to measured exposure or outcome variables, we cannot know if other unmeasured variables affected missingness. However, given the relatively small amount of missing data (6%), any associations of these variables with missingness and score accuracy would have to be very significant to substantially alter our conclusions.
A final potential limitation of our study is that it did not use discharge diagnoses or expert clinician chart review to create a cohort consisting solely of patients with definite sepsis. However, our cohort of patients with suspected sepsis more accurately reflects the population in whom prognostic scores are applied.
qSOFA, SIRS and NEWS, used in isolation, have significant limitations in predicting the outcomes of patients with suspected sepsis at presentation to hospital. However, NEWS is at least equivalent to SIRS and qSOFA across most measure of prognostic accuracy, calling into question the rationale of routinely calculating these sepsis-specific scores in clinical practice. Hospitals and healthcare systems in which NEWS is already routinely recorded should carefully consider whether there is any clinical benefit in adopting qSOFA.
Contributors RG and EN conceived of the study. RG, EN and RM designed the analysis. RG performed the analysis and wrote the first draft of the manuscript. All authors contributed to data collection, interpretation of the data and revision of the manuscript.
Competing interests None declared.
Ethics approval Ethics approval was provided by an NHS Research Ethics Committee (reference 17/NE/0140).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement All Stata code used in the creation of this study is available from the lead author on request.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.