Article Text
PDF
Abstract
Objective We sought to compare the quick sequential organ failure assessment (qSOFA) to systemic inflammatory response syndrome (SIRS), severe sepsis criteria and lactate levels for their ability to identify ED patients with sepsis with critical illness.
Methods We conducted this multicenter retrospective cohort study at five US hospitals, enrolling all adult patients admitted to these hospitals from their EDs with infectious disease-related illnesses from 1 January 2016 to 30 April 2016. We abstracted clinical variables for SIRS, severe sepsis and qSOFA scores, using values in the first 6 hours of ED stay. Our primary outcome was critical illness, defined as one or more of the composite outcomes of death, vasopressor use or intensive care unit (ICU) admission within 72 hours of presentation. We determined diagnostic test characteristics for qSOFA scores, SIRS, severe sepsis criteria and lactate level thresholds.
Main results Of 3743 enrolled patients, 512 (13.7%) had the primary composite outcome. The qSOFA scores were ≥1, >2 and 3 in 1839 (49.1%), 626 (16.7%) and 146 (3.9%) patients, respectively; 2202 (58.8%) met SIRS criteria and 1085 (29.0%) met severe sepsis criteria. qSOFA ≥1 and SIRS had similarly high sensitivity [86.1% (95% CI 82.8% to 89.0%) vs 86.7% (95% CI 83.5% to 89.5%)], but qSOFA ≥1 had higher specificity [56.7% (95% CI 55.0% to 58.5%) vs 45.6% (43.9% to 47.3%); mean difference 11.1% (95% CI 8.7% to 13.6%)]. qSOFA ≥2 had higher specificity than severe sepsis criteria [89.1% (88.0% to 90.2%) vs 77.5% (76.0% to 78.9%); mean difference 11.6% (9.8% to 13.4%)]. qSOFA ≥1 had greater sensitivity than a lactate level ≥2 (mean difference 24.6% (19.2% to 29.9%)).
Conclusion For patients admitted from the ED with infectious disease diagnoses, qSOFA criteria performed as well or better than SIRS criteria, severe sepsis criteria and lactate levels in predicting critical illness.
- qSOFA
- sepsis
- severe sepsis
- critical illness
- emergency department
- Screening
Statistics from Altmetric.com
Key messages
What is already known on this subject
The Society of Critical Care Medicine and the European Society of Intensive Care Medicine have recently proposed the quick sequential organ failure assessment (qSOFA) criteria as replacements for traditional sepsis and severe sepsis criteria to identify critically ill patients outside of the intensive care unit.
Prior examinations of the utility of qSOFA have focused primarily on 30-day mortality, which is not a pragmatic, ED-centred outcome.
What this study adds
In this multicenter study, we evaluated qSOFA from a pragmatic, ED-centred vantage point.
For patients admitted from the ED with infectious disease diagnoses, qSOFA criteria performed as well or better than SIRS criteria, severe sepsis criteria and lactate levels in predicting critical illness.
Background
Early identification of patients who are critically ill or may have poor outcomes is a cornerstone of sepsis patient management. For nearly two decades, clinicians have used systemic inflammatory response syndrome (SIRS), sepsis, severe sepsis and septic shock as the primary terms (with corresponding criteria) to gauge severity of illness and guide patient management. Using these terms and criteria, most EDs have developed screening protocols designed to identify patients with sepsis, especially those who are critically ill and could benefit from prompt therapies, at triage and during early stages of their presentations.
Investigators and experts from the Society of Critical Care Medicine and the European Society of Intensive Care Medicine have recently proposed new criteria to define sepsis.1 They propose that the quick sequential organ failure assessment (qSOFA) criteria, consisting of systolic blood pressure ≤100 mm Hg, respiratory rate ≥22 breaths/min and altered mental status (GCS <15), can be used outside of the intensive care unit (ICU) to identify patients critically ill and in need of intensive care.2Patients are given one point for each criterion and patients with a qSOFA score of >2 are thought to be at high risk for poor outcomes.
Although qSOFA was not expressly developed to replace current ED sepsis screening tools, it nevertheless has been proposed to be used as such.3 4 Several concerns arise when attempting use qSOFA in this manner.5 First, given that qSOFA was developed in a heterogeneous group of patients (including many patients outside of the ED), it may not accurately reflect the ED population. Second, the primary outcome driving the derivation and validation of qSOFA was mortality, which is not the only outcome measure under consideration in ED sepsis screening and disposition decisions. ED clinicians use sepsis screening tools for broader purposes, including identifying the sickest patients among the many they see, initiating critical, time-sensitive interventions and determining the need for ICU care. Finally, the primary outcome of mortality of 30 days is much longer than typical ED-centred outcomes. Patients who may not be critically ill on ED presentation, but subsequently become critically ill after days to weeks in the hospital and die later in the hospital course (or even as outpatients) would be considered to have this outcome, as would patients who receive comfort measures only. In general, triage decisions of ED patient with sepsis are expected to anticipate the need for ICU care and mortality over a much shorter time period after presentation—within approximately 72 hours of presentation.
The objective of this study was to assess the ability of qSOFA to detect patients with critical illness from a pragmatic, ED-centred vantage point. We therefore chose to compare it with the performance of the most commonly used current ED sepsis identification tools of SIRS, sepsis and severe sepsis criteria (former criteria) and lactate levels for predicting the need for critical care and mortality within the first 72 hours of admission to the hospital.
Methods
This retrospective study was conducted at five US hospitals (three urban, county academic medical centres and two community hospitals) between 1 January and 30 April 2016. The mean number of beds of the EDs at these hospitals was 45 (range 24–70) and the mean number of annual visits was 58 700 (range 45 000–72 000). We included adult (age >17 years) patients admitted to an observation unit, inpatient ward or intensive care unit from the ED with a presumed infectious disease-related illness. We excluded patients who (1) were transferred from an outside hospital, (2) admitted to the hospital primarily for other reasons beyond their infectious disease illness (ie, a patient with cellulitis and severe trauma who was primarily admitted for the trauma would be excluded) and (3) left against medical advice (AMA) prior to hospital admission. None of the five hospitals had explicit ICU admission criteria.
Participants
Prior to abstraction of data, we generated an inclusive list of infectious disease-related admission diagnoses (see online Supplementary appendix). Each ED maintains an admission log that lists all admitted patients along with their admission diagnoses. During the study time period, investigators at each site reviewed these admission logs to identify potentially eligible subjects and then reviewed the ED charts of these patients to confirm that they were admitted with diagnoses on this list. As our goal was to test the performance of tools currently used for determining the severity of illness in patients presumed to have sepsis, we included patients who received an infectious disease-related diagnosis in the ED, even if those patients were ultimately found to not have infections during their hospital admission. Ambiguous inclusions or exclusions were adjudicated by consensus review of two additional investigators.
Supplementary file 1
We defined our primary outcome of critical illness as a composite of ICU stay, receipt of vasopressor support or hospital death (excluding hospice or ‘do not resuscitate’ patients) within 72 hours of presentation (any of the three occurring within 72 hours qualified as a positive outcome). Vasopressor support included any intravenous infusions of norepinephrine, epinephrine, dopamine, phenylephrine, dobutamine and vasopressin. Single push-doses of vasopressors were not included.
We adhered to STAndards for the Reporting of Diagnostic accuracy studies (STARD) guidelines for diagnostic test studies and used standardised abstraction methods to record qSOFA, SIRS and severe sepsis criteria from patients’ ED charts and electronic health records.6 7 Specifically, we generated a structured abstraction data collection form, held abstractor orientation meetings and conducted preliminary abstraction of 10 charts with abstractors to ensure abstractor consistency. We also held regular meetings to review progress and conducted frequent audits of abstractor data.
See box 1 for criteria comprising SIRS, sepsis, severe sepsis and qSOFA.8–11 Given that infectious disease-related illness was an inclusion criterion for the study, all patients meeting SIRS criteria also met sepsis criteria and were considered one group: SIRS/sepsis. We used the most abnormal values in the first 6 hours of ED stay for all data points, that is, highest respiratory rate, lowest systolic blood pressure, lowest GCS and highest or lowest temperatures. Because most decisions regarding disposition are made within 6 hours, we did not include vital signs beyond this time point. Respiratory rates at triage were recorded from automated pulse oximetry devices. GCS were obtained from both nursing and physician assessments—at four sites all patients were given GCS scores at triage by nursing. We also recorded first venous lactate levels if performed within 6 hours of ED presentation. Missing data elements were recorded as missing, but assumed to be normal (non-aberrant) in analyses.
Systemic inflammatory response (SIRS), sepsis, severe sepsis and quick sequential organ failure assessment (qSOFA) criteria definitions
SIRS
Two or more of:
Temperature >38°C or <36°C.
Heart rate >90/min.
Respiratory rate >20/min or arterial partial pressure of carbon dioxide <32 mm Hg (4.3 kPa).
White cell count >12×109/L or <4×109/L or >10% immature bands.
Sepsis
Criteria meeting SIRS.
Source of infection.
Severe sepsis=sepsis induced tissue hypoperfusion or organ dysfunction
Criteria meeting sepsis (SIRS+source of infection) and any of the following:
Lactic acidosis ≥2.2 mmol/L.
Urine output <0.5 mL/kg/hour for more than 2 hours despite adequate fluid resuscitation.
Acute lung injury with PaO2/ fractional inspired oxygen (FIO2) <250 in the absence of pneumonia as infection source.
Acute lung injury with PaO2/FIO2 <250 in the presence of pneumonia as infection source.
Creatinine >2.0 mg/dL.
Bilirubin >2 mg/dL.
Platelet count <100 000 µL.
Coagulopathy (international normalised ratio>1.5).
qSOFA
Systolic blood pressure <100 mm Hg.
Respiratory rate ≥22 breaths per min.
Altered mental status (GCS <15).
To minimise bias, we abstracted outcomes blinded to qSOFA, SIRS and sepsis criteria. To measure interabstractor reliability, we conducted dual, independent abstraction of 200 subjects and calculated a kappa statistic of agreement.
Sample size considerations
The sample size to achieve our objectives was governed by the precision (width of (CIs)) around the point estimate of sensitivity of sepsis criteria and qSOFA for the primary composite outcome. From prior studies, we estimated that these instruments would have approximately 85% sensitivity for this combined outcome.12–14 We sought 3% CIs around this point estimate and expected that approximately 15% of enrolled patients would have our primary outcome; we therefore estimated the need to enrol approximately 3533 patients in this study. From pilot data, we estimated that we would enrol approximately 1000 patients per month at the five sites and therefore set our study time period from January to 30 April 2016.
Data management
We managed data using Research Electronic Data Capture hosted by the University of California San Francisco and transferred de-identified data to Excel spreadsheets (Microsoft Office Professional Plus 2010) for sorting and analysis.
Primary outcome analysis
We determined the sensitivity, specificity, positive and negative predictive value and positive and negative likelihood ratios for our composite outcome for qSOFA scores of ≥1, >2 and 3, SIRS/sepsis criteria, severe sepsis criteria and lactate levels of ≥2 and >4. We also generated three receiver operating characteristic (ROC) curves and calculated the area under the ROC curve (AUROC) for these same qSOFA cut-off points, SIRS/sepsis and severe sepsis criteria and lactate level cut-off points. For lactate, we used cut points of 2 and 4, because these cut points are commonly used as thresholds for sepsis interventions. Because the hypotension element in qSOFA could inflate its performance with vasopressor as an outcome and because the qSOFA score could affect decisions about ICU disposition (incorporation bias), we also calculated qSOFA performance for each of the individual elements (ICU stay, mortality, and vasopressor use) as single (non-composite) outcomes.
Results
Of 21 038 patients admitted from the EDs of the study sites, 3869 were deemed to have infectious disease-related illnesses as one of their primary reasons for admission. Subsequent review of 223 ambiguous cases by two authors excluded 126 patients, leaving 3743 patients for full analysis (figure 1). Median age of the cohort was 58 years (IQR (44–72)) and their hospital mortality was 1.9% with a median of 5 days (IQR 2–9) until death. Other patient characteristics are summarised in table 1.
Screening and enrollment.
Patient characteristics, n=3743
Missing qSOFA, SIRS and severe sepsis data elements comprised less than 0.05% of the total; GCS was the most commonly absent criterion of the qSOFA—missing in four (0.1%) of cases. The kappa statistics of agreement with 95% CIs were 0.987 (95% CI 0.97 to 1.00) for qSOFA criteria and 0.972 (95% CI 0.944 to 0.999) for the composite outcome, indicating extremely high and reliable interabstractor agreement.
Among the cohort, 2202 (58.8%) and 1085 (29.0%) met SIRS/sepsis and severe sepsis criteria, respectively. The qSOFA scores were ≥1, >2 and 3 in 1839 (49.1%), 626 (16.7%) and 146 (3.9%) patients, respectively. Of the 2584 patients who had lactate levels measured in the ED, 888 (34.4%) had a lactate level ≥2 and 205 (7.9%) had a lactate level ≥4.
Main analysis
The primary outcome occurred in 512 (13.7%) patients. Among this group, 493 (96.3%) were admitted to an ICU, 219 (42.8%) patients received a vasopressor and 45 (8.8%) patients died within 72 hours of ED presentation; 216 (42.2%) patients had more than one of these outcomes. Mechanical ventilation was delivered to 271 (55.0%) ICU patients in the first 72 hours.
For the primary composite outcome, sensitivity and negative likelihood ratios were highest and nearly identical for qSOFA ≥1 and SIRS/sepsis: sensitivity 86.1% (95% CI 82.8% to 89.0%) versus 86.7% (95% CI 83.5% to 89.5%) and likelihood ratio negative 0.24 (95% CI 0.20 to 0.30) versus 0.29 (95% CI 0.23 to 0.36). However, qSOFA ≥1 had greater specificity than SIRS/sepsis [56.7% (95% CI 55.0 to 58.5%) vs 45.6% (95% CI 43.9% to 47.3%); mean difference 11.1% (95% CI 8.7% to 13.6%)]. A qSOFA ≥2 had higher specificity than severe sepsis criteria [89.1% (95% CI 88.0% to 90.2%) vs 77.5% (95% CI 76.0% to 78.9%); mean difference 11.6% (95% CI 9.8% to 13.4%)] at the expense of lower sensitivity [mean difference 16.2% (95% CI 10.3% to 22.0%). A qSOFA=3 had the highest specificity but lower sensitivity than SIRS/sepsis or severe sepsis. A lactate level ≥2 had much lower sensitivity than qSOFA ≥1 (mean difference 24.6% (95% CI 19.2% to 29.9%)]. Like qSOFA=3, a lactate level ≥4 had high specificity but very low sensitivity (see table 2).
qSOFA, SIRS, severe sepsis and lactate level: screening performance characteristics for composite outcome
ROC curves are presented in figure 2; qSOFA had significantly higher AUROC (0.788) than SIRS/sepsis or severe sepsis criteria (0.754) or lactate level thresholds (0.763), p=0.0026 and p<0.001, respectively.
ROC for qSOFA, lactate and SIRS/Severe Sepsis Criteria.
Sensitivity analyses
Removal of the vasopressor criterion from the composite outcome had very little effect on qSOFA performance. A qSOFA score of ≥1 had a sensitivity of 86.6% (95% CI 83.2% to 89.4%) and a specificity of 56.6% (95% CI 54.9% to 58.3%) for the composite outcome of ICU admission or mortality. A qSOFA score of ≥2 had a sensitivity of 54.0% (95% CI 49.5% to 58.4%) and a specificity of 89.0% (95% CI 87.9% to 90.0%).
In tables 3–5, we present the performance characteristics of qSOFA for predicting the single outcomes of mortality, vasopressors and ICU admission within 72 hours, respectively. For all of these outcomes, a qSOFA score of 1 had high sensitivity (>84%) and low specificity (<57%), and a qSOFA score of 3 had low sensitivity (<43%) and high specificity (<96%).
qSOFA: screening performance characteristics for mortality at 72 hours
qSOFA: screening performance characteristics for vasopressors at 72 hours
qSOFA: screening performance characteristics for intensive care unit at 72 hours
Discussion
Identifying patients with critical illness is paramount in the ED care of patients with infectious disease processes. Highly sensitive prediction tools will allow clinicians to detect all (or nearly all) critically ill patients early in their course to provide time-sensitive interventions such as fluid resuscitation, early antibiotics and timely vasopressor administration.15–18 Highly specific predictive tools allow for more judicious disposition decisions and expenditure of resources, particularly ICU beds. In this multicenter study assessing the utility of qSOFA to predict the need for ICU care, vasopressors or mortality within 72 hours of admission from the ED, we found that qSOFA criteria had slightly better performance than traditional SIRS and sepsis criteria. For ruling out critical illness, qSOFA ≥1 had equivalent sensitivity and negative predictive value to SIRS criteria, with slightly higher specificity and positive predictive value. For ruling in critical illness, we found that a qSOFA ≥2 offers superior discriminative performance (higher specificity and positive likelihood ratio) for short-term critical illness than formerly used SIRS and severe sepsis criteria. When combining overall prognostic accuracy in terms of ROC curve comparisons and AUROC, we found that qSOFA had better discriminant capacity than SIRS/severe sepsis criteria and lactate levels.
Complicated formulas and instruments requiring extensive laboratory testing are less useful than simple ones from a practical implementation standpoint in the ED. With only three very straightforward clinical criteria that are easily assessed in all ED patients and no need for laboratory testing, qSOFA as a risk-stratification tool has substantial pragmatic appeal compared with the criteria for SIRS, sepsis and severe sepsis criteria.
Evaluations of qSOFA have not specifically addressed the utility of qSOFA for identifying patients who will have critical outcomes in the first 72 hours after admission from the ED.2 11–14 19 20 These studies have nevertheless yielded mixed results. Most investigators have found that qSOFA was a better predictor of 30-day mortality than SIRS or severe sepsis criteria.2 11–14 19 However, other investigators found that although qSOFA and traditional SIRS criteria had similar discrimination for predicting organ dysfunction, qSOFA ≥2 was much less sensitive for 30-day mortality than SIRS.20
Given recent concerns of inadequate sensitivity (87.9%) of SIRS criteria for critical illness, the 86.1% sensitivity of a qSOFA ≥1 may seem insufficient.21 In this regard, other risk stratification tools for critical illness have been proposed as useful alternatives to current standard critical illness screening tools.21 The National Early Warning Score (NEWS) and Modified Early Warning Score (MEWS) criteria, which also consist of readily available physiological parameters, have performed well when used to identify critical illness in heterogeneous patient populations.22–24 In a single-site validation study, Churpek et al, found that both NEWS and MEWS were more accurate than qSOFA ≥2 at predicting early mortality and need for ICU transfer.13
Despite the initial reluctance of emergency physicians to embrace qSOFA as a substitute for SIRS, sepsis and severe sepsis criteria, we believe our findings are in line with those presented by the Sepsis-3 task force and that qSOFA may be a superior indicator of critical illness in these patients. As with all scoring systems, we believe that qSOFA should not replace clinical judgement, but should aid clinicians in both risk stratification and clinical decision-making. Given the superior performance over previous methods of identifying patients with infectious disease who are critically ill, the early use of qSOFA may help improve the identification of those who need timely interventions, further diagnostic testing and potentially higher intensity care. Future investigations should assess these tools prospectively.
Limitations
Although our study design was retrospective, less than 1% of our eligible subjects had missing qSOFA or outcome data and we had excellent high interabstractor agreement. The study only included patients admitted to the hospital and not those discharged, and thus did not assess qSOFA’s ability to detect critical illness among all potentially infected patients. Although our use of admission logs and ED diagnoses to screen for eligible patients could lead to selection bias, this method is ED centred and practical. Use of blood cultures as a required inclusion criterion would unnecessarily exclude many patients who may still be septic. For example, current Infectious Disease Society of America guidelines recommend against routine blood cultures in patients with skin and soft-tissue infections.25
Incorporation bias may have affected the predictive performance of SIRS and sepsis criteria, as well as lactate levels, for the composite outcome by virtue of the fact that these criteria may have been used by clinicians in real time to determine the need for ICU care. Because we examined data from patients before the publication of the qSOFA score, SIRS and severe sepsis screening performance would thereby likely be inflated to a greater extent than qSOFA. Given that one of the qSOFA criteria is hypotension (systolic blood pressure <100), our inclusion of vasopressor use as part of our composite outcome may have biased our results in favour of qSOFA. However, when removing this outcome, qSOFA retained superior predictive performance. Sensitivity analyses of individual outcomes (mortality alone, vasopressor alone) demonstrated similar test characteristics to the composite outcome. Although the original qSOFA cut-off point was 2, our intent was to analyse outcomes according to the full spectrum of qSOFA scores. Dichotomisation of lactate may have also decreased its discriminatory ability. We found lower overall hospital mortality than that seen in the qSOFA study cohorts1 2; however, this was an expected finding given that we included all patients admitted from the ED rather than only patients who had ICU stays.
Conclusions
In this multicenter study of patients admitted with infectious disease diagnoses, we found that qSOFA criteria had better performance for predicting early critical illness than SIRS, sepsis or severe sepsis criteria or lactate levels. Given that qSOFA criteria are easy to perform without the need for blood tests, they may be preferred over current ED risk stratification tools in the ED.
References
Footnotes
Contributors RMR, JCG, TJN, BD and BMB contributed to the study design. RMR, JCG, BD, TJN, BMB and FSS were involved in the data analysis. All authors contributed to the study implementation/data acquisition, manuscript preparation and revision.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent Not required.
Ethics approval IRBs of UCSF, University of Penssylvania and Sutter.
Provenance and peer review Not commissioned; externally peer reviewed.
Linked Articles
- Commentary
- Primary survey