Article Text
PDF
Abstract
Objective Validated clinical risk scores are needed to identify patients with COVID-19 at risk of severe disease and to guide triage decision-making during the COVID-19 pandemic. The objective of the current study was to evaluate the performance of early warning scores (EWS) in the ED when identifying patients with COVID-19 who will require intensive care unit (ICU) admission for high-flow-oxygen usage or mechanical ventilation.
Methods Patients with a proven SARS-CoV-2 infection with complete resuscitate orders treated in nine hospitals between 27 February and 30 July 2020 needing hospital admission were included. Primary outcome was the performance of EWS in identifying patients needing ICU admission within 24 hours after ED presentation.
Results In total, 1501 patients were included. Median age was 71 (range 19–99) years and 60.3% were male. Of all patients, 86.9% were admitted to the general ward and 13.1% to the ICU within 24 hours after ED admission. ICU patients had lower peripheral oxygen saturation (86.7% vs 93.7, p≤0.001) and had a higher body mass index (29.2 vs 27.9 p=0.043) compared with non-ICU patients. National Early Warning Score 2 (NEWS2) ≥ 6 and q-COVID Score were superior to all other studied clinical risk scores in predicting ICU admission with a fair area under the receiver operating characteristics curve of 0.740 (95% CI 0.696 to 0.783) and 0.760 (95% CI 0.712 to 0.800), respectively. NEWS2 ≥6 and q-COVID Score ≥3 discriminated patients admitted to the ICU with a sensitivity of 78.1% and 75.9%, and specificity of 56.3% and 61.8%, respectively.
Conclusion In this multicentre study, the best performing models to predict ICU admittance were the NEWS2 and the Quick COVID-19 Severity Index Score, with fair diagnostic performance. However, due to the moderate performance, these models cannot be clinically used to adequately predict the need for ICU admission within 24 hours in patients with SARS-CoV-2 infection presenting at the ED.
- emergency department
- intensive care
- COVID-19
- risk management
Data availability statement
Data may be obtained from a third party and are not publicly available. We used data from the ongoing COVIDPredict Clinical Course Cohort. This is a nationwide database. Data may be used exclusively by scientists from the included hospitals.
This article is made freely available for use in accordance with BMJ’s website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may use, download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.
https://bmj.com/coronavirus/usageStatistics from Altmetric.com
Key messages
What is already known on this subject
The SARS-CoV-2 infection that causes COVID-19 poses multiple challenges to healthcare systems worldwide. A particular challenge is that a significant proportion of patients requires transfer to intensive care units because of respiratory failure. However, the effectiveness of early warning scores for predicting deterioration for patients with COVID-19, remains unknown.
What this study adds
The National Early Warning Score 2 and the Quick COVID-19 Severity Index Score have fair diagnostic performance in predicting deterioration of patients with COVID-19 at the ED; however due to their moderate performance, these models cannot be adequately used to predict critical illness.
Introduction
Background
The SARS-CoV-2 infection that causes COVID-19 poses multiple challenges to healthcare systems worldwide. A particular challenge is that a significant proportion of patients requires transfer to intensive care units (ICU) because of respiratory failure. In the current pandemic situation, early and effective identification and triage of patients at risk of severe disease/deterioration at the ED is crucial.
To recognise a patient’s clinical deterioration, several early warning scores (EWS) have been developed and validated.1 The National Early Warning Score 2 (NEWS2);2 Modified Early Warning Score (MEWS);3 the quick Sequential Organ Failure Assessment (qSOFA);4 the systemic inflammatory response syndrome (SIRS) criteria5 and Confusion, Urea nitrogen, Respiratory rate, Blood pressure and 65 years or older (CURB-65)6 are the most fitting models as they are validated for a short-term clinical outcome. The authors of a recent study concluded that NEWS2 may adequately predict ICU admission in patients with COVID-19 at the ED, however the study population was very small.7 In addition, a COVID-19-specific prognostic tool called the Quick COVID-19 Severity Index (qCSI) has been recently developed. However, this COVID-19-specific prognostic tool has not been externally validated yet.8
According to prior research, it was expected that qCSI and NEWS2 could adequately identify which patients with COVID-19 are at risk for requiring ICU admission with an area under the receiver operating characteristics (AUROC) curve of at least 0.70.7 8
Importance
The COVID-19 pandemic is causing overcrowded EDs and leads to scarcity in medical personnel, medical equipment and hospital beds. Therefore, effective and easy-to-use risk-stratification tools are needed to identify patients that are at high risk of clinical deterioration and need admission to inhospital wards or, in particular, to ICUs.
Goals of this investigation
The aim of this study was to assess and compare the performance of several commonly used clinical risk scores in their ability to identify which patients with COVID-19 are likely to deteriorate and will require ICU care within 24 hours after ED presentation defined as needing high-flow-oxygen usage or mechanical ventilation.
Methods
Study design, population and setting
We used data from the ongoing COVIDPredict Clinical Course Cohort containing over 2000 patients with COVID-19 who were recruited in nine hospitals in the Netherlands (of which seven are general and two are university hospitals). Included in the database were all subjects admitted to the hospital between 24 February and 31 July 2020, with a positive result of a SARS-CoV-2 PCR collected at the nasopharynx, the throat, from sputum or from a bronchoalveolar lavage sample or a CT scan with a COVID-19 Reporting and Data System (CO-RADS) Score of 4 or more. CO-RADS is a categorical assessment scheme for pulmonary involvement of COVID-19 at unenhanced chest CT that performs very well in predicting COVID-19 in patients.9 Exclusion criteria were patients with do-not-resuscitate orders and interhospital transferred patients.
Patient and public involvement
No patient was involved.
Data collection
The data extracted from the COVIDPredict database included baseline characteristics such as age, sex and a full set of vital variables including respiratory rate, peripheral oxygen saturation (SpO2), heart rate, blood pressure and GCS at presentation at the ED. Additionally, the following interventions were recorded: administration of oxygen (yes/no); including mode of delivery (nasal cannula, non-rebreathing masks, continuous positive airway pressure masks, trachea intubation). To reduce bias, we used single imputation with normal value substitution for missing values needed to calculate the total risk scores.
Employed scoring systems
NEWS2, MEWS, qSOFA and CURB-65 are all risk models developed to identify patients likely to deteriorate; deterioration is often defined as death, severe adverse event or admission to the ICU. They are based on aggregated scoring of physiological parameters, the usage of supplementary oxygen and patient characteristics, see (table 1). Each model has its own maximum sumscore and its threshold point (cut-off point) for optimal usage. qCSI is a similar risk model as NEWS2, however it includes less vital parameters and is developed specifically to predict deterioration of patients suffering from SARS-CoV-2.8
Early Warning Score characteristics
Outcomes
The primary outcome was the model performance of NEWS2, qCSI, MEWS, qSOFA, SIRS criteria and CURB-65 risk model to discriminate patients needing ICU admittance within 24 hours after presentation at the ED.
Sample size analysis
The sample size was calculated with a two-sided 95% CI for a single proportion with a fixed margin of error of 2.5%. According to a systematic review, the observed proportion of ICU admission was 0.26.10 Using these numbers this resulted in a needed sample size of 1183 patients.
Data analyses
Recorded data were presented as absolute values with percentages and continuous data as mean values with SD or median values with IQRs, depending on whether data were normally distributed. Normal distribution of variables was assessed with Kolmogorov-Smirnoff tests as well as exploring frequency distributions (histograms). Numerical variables with a normal distribution were evaluated using the Student’s t-test or the Mann-Whitney U test in case there was no normal distribution. χ2 analyses were used for statistical testing of categorical data.
Our hypothesis was tested by means of assessing model accuracy using a specified reference value (ie, AUROC curve) with binomial 95% CIs. In addition to the AUROC, sensitivity and specificity were calculated for the key thresholds.
The AUROC resulted from the different risk scores in predicting the need of ICU admission were calculated and the performance was compared using the method of Hanley and McNeil.11 All analyses were performed with SPSS (V.23.0; SPSS, Chicago, Illinois, USA). All tests of significance used a two-sided p<0.05.
Results
Between 24 February and 31 July 2020, a total of 2338 patients was admitted to one of the nine participating hospitals. All patients had a positive SARS-CoV-2 PCR test or a CO-RADS of 4 or higher confirmed by a radiologist. Of these patients, 242 were excluded as they were transferred from other hospitals and 595 were excluded as they had a ‘do not resuscitate’ order. This resulted in a study group of 1501 patients included in the analysis.
The median age was 71 years (range 19–99 years), and 905 out of 1501 patients (60.3%) were male. In total, 1304 patients (86.9%) were admitted to the ward and 197 (13.1%) to the ICU within 24 hours after initial presentation at the ED. Compared with the non-ICU patients, the ICU patients were more often male (75.6% vs 50.4%, p≤0.01), had lower SpO2 (86.7% vs 93.7, p=<0.001), were more tachypnoeic (28.8 vs 22.4 breaths/min, p≤0.001) and had a higher body mass index (29.2 vs 27.9 p=0.043) compared with the non-ICU patients (see table 2).
Patient characteristics
No vital parameters were missing to calculate the investigated risk scores. However, leucocyte count was missing for 762 patients.
In total, 889 patients had NEWS2 ≥ 5 and 720 patients had NEWS2 ≥ 6 at admission. NEWS2 ≥ 6 was superior to the other studied clinical risk scores with an AUROC of 0.740 (95% CI 0.696 to 0.783) (see table 3), in predicting critical illness within 24 hours after ED presentation. The qCSI risk model had an AUROC of 0.760 (95% CI 0.719 to 0.800), comparable to that of NEWS2 (p=0.25) (see figure 1).
Clinical risk scores and their area under the receiver operating characteristics (ROC). CRB, Confusion, Urea nitrogen Respiratory rate and Blood pressure, NEWS2, National Early Warning Score 2; MEWS, Modified Early Warning Score; qCSI, Quick COVID-19 Severity Index; qSOFA, Quick Sequential Organ Failure Assessment.
Clinical risk scores and their area under the receiver operating characteristics (AUROC) curve
Based on the AUROC of the NEWS2, sensitivity and specificity were calculated for a score ≥5. We observed a sensitivity and specificity of 84.7% (95% CI 78.9% to 89.4%) and 44.3 (95% CI 41.5% to 47.0%), respectively. With a threshold score of ≥6, sensitivity decreased to 78.1% (71.6–83.6) and specificity increased to 56.3% (53.5–59.0).
Discussion
In the current COVID-19 pandemic, early identification and triage of patients at risk of severe disease requiring ICU admittance is crucial as hospital resources are scarce. Many EWS have been developed to identify the deteriorating patient. The main finding of the current study was that the predictive power of NEWS2 ≥ 6 to discriminate patients at the ED to be submitted to the regular ward or to the ICU is comparable for the qCSI score with fair AUROC of 0.740 and 0.760, respectively. NEWS2 and qCSI score are both superior to the other investigated risk scores. However, qCSI consists of less physiological parameters compared with NEWS2 and may therefore be easier to use. However, the performance of the best performing models is moderate. Due to the relatively poor performance, these models cannot be clinically used to adequately predict the need for ICU admission for the investigated study population.
NEWS2 ≥6 discriminated patients needing ICU admission with 78.1% sensitivity and 56.3% specificity. A qCSI score ≥3 at admission discriminated patients needing ICU admission with 75.9% sensitivity and 61.8% specificity. While sensitivity of both models is high and the number of false-negative results is therefore expected to be relatively low, specificity is moderate, resulting in potential false positives.
EWS may be valuable as they can assist the medical staff in estimating the chance of the patient’s clinical deterioration and empower escalation of medical care in forms of more intensive treatment or continued monitoring of vital parameters, which improves patients outcome.12 13 However, since EWS are developed to aid clinical decision-making and the model performance is relatively poor, they should not be used as a barrier to, or an alternative for, skilled clinical judgement.2
The qCSI is a recently developed model to predict which patients needing supplemental oxygen at the ED are likely to deteriorate. The original qCSI study excluded patients who required >6 L O2 administration in the ED, as these patients were likely to deteriorate and thus required ICU admission. However, one of the main reasons to use an EWS is that it should be easy to use and be applicable to a wide variety of patients. Therefore, we did not exclude patients needing >6 L O2 in our analysis.
Strengths and limitations
The main strength of our study is our large study cohort, including consecutive patients with COVID-19 admitted to nine teaching hospitals in the Netherlands from the start of the outbreak. Also, all vital signs needed for the EWS were complete and available at ED admission, with no missing data and reducing any potential selection bias. However, leucocyte count, used in calculating the SIRS criteria, was missing for 762 patients and therefore had to be imputed. Missing data were imputed as normal values, leading to a potential concerning bias as normal values may not have been underestimated. As SIRS performed significantly worse compared with the other models, extra sensitivity analysis using multiple imputations with chained equities was not performed.
Another limitation is that although vital parameters were all measured at the ED, the timing of the measurements is unclear. In addition, due to the nature of the study design, unstructured provider judgement is missing, therefore, it remains unknown whether the investigated tools have utility beyond clinical gestalt. Further research should investigate the effectiveness of EWS in addition to clinical gestalt for predicting the need for ICU admission.
Data of patients not being admitted to the hospital 24 hours after ED admission, but who were discharged directly form the ED, were not included in the database. Therefore, selection bias may have been introduced. However, our results are comparable with prior literature as the AUROC of NEWS2 is often around 0.750 in the average ED population.14
The median age of the ICU population was significantly lower compared with the non-ICU patients. In the Netherlands the first part of the pandemic led to strict selection of patients (based on age, Clinical Frailty Score and comorbidities) for ICU admittance. Therefore, 25.5% of the patients were excluded as they had a do-not-resuscitate order. Due to the shortage of hospital beds in the first wave of the pandemic, the do-not-resuscitate topic was often discussed with the frail and elderly. Therefore, the high amount of do-not-resuscitate patients may be explained. This is in concordance with prior research, that demonstrated that in the first wave of COVID-19, patients accepted for ICU treatment were younger and less frail than those previously accepted for ICU treatment.15
During the first months of the pandemic, no specific treatment guide was present. Therefore, many patients with low pulse oxymetry in combination with high supplemental oxygen usage were transferred to the ICU as early intubation was recommended to avoid prolonged non-invasive ventilation and to reduce aerosol exposure for the medical staff. However, with dexamethasone as a treatment and evidence that high-flow nasal cannula may reduce the need for invasive ventilation and ICU admission, the investigated patient group may differ from the patients currently admitted to the ICU with COVID-19.16 17
In this multicentre study, the best performing models to predict ICU admittance were NEWS2 and the qCSI Score, with fair diagnostic performance. However, due to the relatively poor performance, these models cannot be clinically used to adequately predict the need for ICU admission within 24 hours in patients with SARS-CoV-2 infection presenting at the ED.
Data availability statement
Data may be obtained from a third party and are not publicly available. We used data from the ongoing COVIDPredict Clinical Course Cohort. This is a nationwide database. Data may be used exclusively by scientists from the included hospitals.
Ethics statements
Patient consent for publication
Ethics approval
AmsterdamUMC approved and developed the COVIDPredict database and approved the writing of this manuscript. The ethics committee of the Amsterdam-UMC location AMC approved this research (waiver W20_391 20.435).
Acknowledgments
The authors thank the COVIDPredict Study Group for contribution towards the data collection.
References
Footnotes
Handling editor Katie Walker
Twitter @SchinkelMichiel
Contributors JB, MB, TD, RD, NG, LH, KK, MK, PN, AR, WR, SS, CW collected the data. LV designed, analysed, interpreted the data and draft the article. ML, MS, JB, MB, RD, SS, CW, PN, MH revised critically for important intellectual content and helped with the final version to be submitted. LV is the gauranter of this study.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.