Article Text
Statistics from Altmetric.com
Liang and colleagues developed a risk prediction score, COVID-GRAM, to identify adults with COVID-19 at higher risk of intensive care stay, mechanical ventilation or death.1 This score had strong performance in Chinese cohorts and has been validated in multiple non-US cohorts, although with variation in its performance (C-statistic ranging from 0.64 to 0.91).1 2 It has yet to been studied in US populations.1 2 Differences in the US hospital practices and patient population may affect the applicability of COVID-GRAM to this population. Additionally, clinical rationale and prior studies suggest that CURB-65 may predict severe disease in COVID-19.3 We compare the performances of COVID-GRAM with CURB-65 for predicting critical illness in patients with COVID-19 in a US population.
This retrospective study included adult patients admitted to an academic medical centre in Boston Massachusetts with a diagnosis of COVID-19 between 1 January 2020 and 29 June 2020. Individuals with prior COVID-19 hospitalisations were excluded. Patients were followed until outcome occurrence or the end of hospitalisation (whichever came first). Demographic and clinical data, patient outcomes and variables used in COVID-GRAM and CURB-65 were obtained from the electronic health record. The primary outcome was critical illness—defined as a composite of mechanical ventilation or death. We used multivariable logistic regression to determine the association between predictors and the outcome of critical illness. Two models of critical illness were constructed: one with all predictors in COVID-GRAM and one with all predictors in CURB-65, with the model coefficients matching those in the original risk scores. The predictive ability of each scoring system was determined using C-statistic, and predictive performance between the two scores was assessed using the de-Long test. Missing data were accounted for using multiple imputations. We used five imputations each to determine the values of the missing data. We chose this approach due to <20% missingness for each predictor. In sensitivity analyses, we repeated the analytical approach but as a complete case analysis. A p value of 0.05 was considered statistically significant. Analyses were conducted with R V.3.5.2.
The study was approved by the Beth Israel Deaconess Medical Centre institutional review board and determined to be exempt. No patients were directly involved in our study.
Of 844 patients presenting to the ED, 546 were admitted (figure 1). The mean age was 66.8 years (SD: 16.9 years), of whom 48.5% were women (table 1). The primary composite outcome occurred in 170 individuals. Due to missing data for score calculation (131 individuals with missing data for COVID-GRAM calculation and 51 with missing data for CURB-65 calculation), 495 individuals were included in the primary analysis. Increasing score with COVID-GRAM was associated with critical illness (p<0.001). COVID-GRAM had modest discrimination (C-statistic=0.72 (95% CI: 0.67 to 0.76)) (figure 2). CURB-65 score was also associated with critical illness (p<0.001). However, discrimination of CURB-65 (C-statistic: 0.61 (95% CI: 0.56 to 0.66)) was lower than COVID-GRAM (p<0.001). The association of each predictor with the outcome of interest is demonstrated in online supplemental file 1. In our complete case sensitivity analyses, COVID-GRAM (C-statistic: 0.70 (95% CI: 0.65 to 0.75), p<0.001) and CURB-65 (C-statistic: 0.58 (95% CI: 0.53 to 0.64), p=0.003) had similar performance to the primary analysis. Discrimination of CURB-65 was lower than COVID-GRAM (p<0.001).
Supplemental material
There are several limitations to our study. Small cohort size may have prevented the identification of certain associations with severe disease. Total bilirubin was used instead of direct bilirubin. Our study was conducted at a single academic medical centre and could not assess the effects of different COVID-19 strains. Because we focused on outcomes occurring during the index hospitalisation, we may have failed to capture rehospitalisation with critical illness.
In this US hospital, COVID-GRAM had modest accuracy in identifying patients who were likely to require mechanical ventilation or expire and substantially outperformed CURB-65. Although COVID-GRAM incorporates predictors routinely obtained in clinical settings and is superior to CURB-65, we believe that the moderate discrimination of COVID-GRAM means that it should not be used in isolation for risk prediction, but rather as an adjunct to clinical reasoning.
Ethics statements
Patient consent for publication
Ethics approval
The study was approved by the Beth Israel Deaconess Medical Centre institutional review board and determined to be exempt.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Handling editor Richard Body
Twitter @RahulAggarwalMD
Contributors RA, TSA, JPS, SJH conceptualised the idea. AM, AP, AA, NP, MD, TM, TL, NF were involved in data collection. RA was involved in data analytics. LN was involved in analytical planning and statistical guidance. JPS and SJH supervised the study. All authors were involved in drafting the manuscript, intellectual design and critical revision of the manuscript for intellectual content.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests TSA discloses consulting fees from Alosa Health.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.