Article Text
PDF
Abstract
Background COVID-19 symptoms vary widely. This retrospective study assessed which of three clinical screening tools—a nursing triage screen (NTS), an ED review of systems (ROS) performed by physicians and physician assistants and a standardised ED attending (ie, consultant) physician COVID-19 probability assessment (PA)—best identified patients with COVID-19 on a subsequent reverse transcription PCR (RT-PCR) confirmation.
Methods All patients admitted to Boston Medical Center from the ED between 27 April 2020 and 17 May 2020 were included. Sensitivity, specificity and positive predictive value (PPV) and negative predictive value (NPV) were calculated for each method. Logistic regression assessed each tool’s performance.
Results The attending physician PA had higher sensitivity (0.62, 95% CI 0.53 to 0.71) than the NTS (0.46, 95% CI 0.37 to 0.56) and higher specificity (0.76, 95% CI 0.72 to 0.80) than the NTS (0.71, 95% CI 0.66 to 0.75) and ED ROS (0.62, 95% CI 0.58 to 0.67). Categorisation as moderate or high probability on the ED physician PA was associated with the highest odds of having COVID-19 in regression analyses (adjusted OR=4.61, 95% CI 3.01 to 7.06). All methods had a low PPV (ranging from 0.26 for the ED ROS to 0.40 for the attending physician PA) and a similar NPV (0.84 for both the NTS and the ED ROS, and 0.89 for the attending physician PA).
Conclusion The ED attending PA had higher sensitivity and specificity than the other two methods, but none was accurate enough to replace a COVID-19 RT-PCR test in a clinical setting where transmission control is crucial. Therefore, we recommend universal COVID-19 testing prior to all admissions.
- COVID-19
- clinical assessment
- diagnosis
- infections
Data availability statement
No data are available.
This article is made freely available for personal use in accordance with BMJ’s website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may use, download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.
https://bmj.com/coronavirus/usageStatistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Symptom screeners for COVID-19 have not been extensively studied as a method for cohorting patients admitted to the hospital from the ED.
This study was performed to assess how useful screening for specific symptoms and travel history was in determining which patients ultimately tested positive for COVID-19.
WHAT THIS STUDY ADDS
In this retrospective study of all patients aged 18 years or older who were admitted to a major urban hospital from the ED from 27 April 2020 to 17 May 2020, none of the three screening methods were sensitive or specific enough to be performed in lieu of a COVID-19 reverse transcription PCR test on a patient ill enough to be admitted to the hospital.
Although fever, chills, fatigue, sore throat, rhinorrhoea and cough were associated with higher odds of having COVID-19, there were many patients with COVID-19 who presented with no or non-specific symptoms that still warranted testing, particularly given that community prevalence of COVID-19 was high during the study period.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE, OR POLICY
In high transmission areas, hospitals should not rely on symptoms or probability status in determining infection status, and should continue to use widespread diagnostic testing.
Introduction
As of 23 December 2022, over 650 million COVID-19 cases and over 6.6 million deaths have been reported worldwide.1 COVID-19 symptoms vary widely, and many are non-specific.2–4 The incidence of asymptomatic infection is unknown but ranges from 1.6% to 56.5% in various studies.5 It is crucial to identify patients with COVID-19 early in their hospitalisation to prevent the spread of infection. Therefore, there is an opportunity to use clinical screening methods when patients present to care to try to predict which patients will test positive for COVID-19 to enable cohorting of admitted patients.
Early in the COVID-19 pandemic, diagnostic testing was not always available, and there was a role for clinical decision methods in identifying patients more or less likely to have COVID-19. One study done in Washington, DC incorporated demographic data, comorbidities, symptoms and objective data to develop a COVID-19 prediction score and found that certain characteristics (eg, nursing facility residence) were associated with having COVID-19.6 However, while the study authors suggested that a risk prediction score for COVID-19 could be used to help providers determine which patients admitted from the ED to the hospital required COVID-19 testing and isolation, the study authors ultimately recommended that all admitted patients be universally tested for COVID-19 since the sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of the tool varied based on the assigned prediction score. Another study in Belgium that evaluated a clinical decision tool based on 11 symptoms using data from March through June 2020 found that patients with fever and dry cough were more likely to have COVID-19, while those with chest pain and sore throat were less likely to have COVID-19.7 However, the tool’s sensitivity was only 61%–62% and its specificity was 70%–74%, and it was not specifically designed to be used to triage patients being admitted to the hospital from the ED. Per our review, there have not been studies that have simultaneously compared the performance of multiple methods used to predict the likelihood of a patient having COVID-19 in the ED.
During early April 2020, 70% of patients admitted to Boston Medical Center (BMC) had COVID-19, many of whom required treatment in the intensive care unit.8 The goal of this study was to retrospectively assess which of the three methods—a nursing triage screen (NTS) using US Centers for Disease Control and Prevention (CDC) screening questions, an ED review of systems (ROS) performed by physicians and physician assistants or a standardised COVID-19 probability assessment (PA) by an ED attending (ie, consultant) physician—was better at detecting patients who ultimately tested positive for COVID-19.
Methods
Study setting
BMC is a 514-bed urban, academic hospital in Boston, Massachusetts. It is the largest safety net hospital in New England, with over half of patients coming from underserved populations, and has over 130 000 ED patient visits per year.9 Over 70% of patients are from racial and ethnic minority populations, 32% do not speak English as a primary language and over 50% rely on Medicare or Medicaid.9
Inclusion criteria
This retrospective study included all patients aged 18 years or older who were admitted to BMC from the ED from 27 April 2020 to 17 May 2020. This represents the 3-week period after the start of routine COVID-19 reverse transcription PCR (RT-PCR) testing for all patients being admitted, and all patients included therefore had PCR tests. Screening assessment methods were analysed if they were present in the medical record, and 531 (70.9%) of patients had all three screening assessments performed.
For patients with multiple admissions during the study period, only the first visit was included. For visits with several COVID-19 test results, the most recent results prior to or during admission were used.
Data collection
Data were obtained from BMC’s electronic medical record (EMR) system through manual chart review and automated data abstraction. Data collected included patient age, sex, race/ethnicity, preferred language, COVID-19 test result(s), NTS results, the ED ROS and the ED attending physician PA.
Interpretation of COVID-19 test results
When this study was done, three COVID-19 PCR tests were available for use at BMC—the Simplexa DiaSorin Molecular test, the Roche Cobas test and the GeneXpert Cepheid test. The test turnaround time was 1–4 hours depending on ED demand and the instrument used. The Simplexa test has a sensitivity of 100% and a specificity of 100%, the Roche test has a sensitivity of 87% and a specificity of 97% and the Cepheid test has a sensitivity of 100% and a specificity of 100%.10–12
According to BMC clinical protocol at the time, patients were assigned a PA of no concern, or low, moderate or high probability for COVID-19. The reference standard for determining the presence or absence of COVID-19 depended on the PA. Those with no concern or low probability were deemed negative after one negative test. Moderate or high probability patients were deemed negative only if two tests within 48 hours of each other were negative, and deemed negative but unconfirmed if they underwent only one test which was negative. Any one positive test was deemed positive for COVID-19. Those with an inconclusive test result (defined as some number of targets of the assay not detected at the level above positive threshold) were treated as presumptive positive. For this analysis, if the first test was inconclusive, unknown or pending, the results of a second test were considered if the test was ordered within 48 hours of the first test. If the second test was inconclusive, unknown or pending, the results of a third test were considered if the test was ordered within 48 hours of the second test. COVID-19 test results that did not have a corresponding admission date were excluded.
Screening methods
Nursing triage screen
The systematic NTS, which was integrated into the EMR, represented the first point of contact between a patient and a healthcare professional. On arrival to the ED, a nurse asked patients whether they had travelled internationally or had been in contact with someone with confirmed or suspected COVID-19 in the past month, and if they had any of the following: fever, chills, weakness, severe headache, anosmia, dysgeusia, conjunctival injection, sore throat, cough, shortness of breath (SOB), abdominal pain, vomiting, diarrhoea, bruising or bleeding, myalgia, arthralgia and rash. If ‘no symptoms’ was selected, the response to each individual symptom was assumed to be no. If any symptom was checked, the response to any remaining unchecked symptoms was assumed to be no. If nothing was checked or if ‘unable to assess’ was selected, responses to all symptoms, including ‘no symptoms’, were assumed to be missing. If there were conflicts (eg, ‘no symptoms’ and ‘fever’ were checked), then all values were deemed as missing. The NTS was positive if it met a modified version of the clinical symptom criteria from the CDC case definition of COVID-19 during that time: either (a) ≥1 of the following: cough or SOB, or (b) ≥2 of the following: fever (measured or subjective), chills, headache or sore throat (figure 1).13
Criteria required for positive nursing triage screen (NTS) and positive ED provider review of systems (ROS).
ED provider ROS
Patients were treated by physician assistants, resident physicians from several departments (emergency medicine, family medicine, internal medicine and obstetrics and gynaecology) and attending emergency medicine physicians. As part of standard evaluation, an ROS was performed. For this study, the ROS was abstracted from the standard ROS performed using a smart form in BMC’s EMR during patient visits and included fever, chills, fatigue, headache, anosmia, dysgeusia, sore throat, rhinorrhoea, cough, SOB, chest pain, abdominal pain, anorexia, nausea, vomiting, diarrhoea and myalgias. If the ROS was not performed or was incomplete, a reason was documented, such as altered mental status, dementia, intoxication, or non-verbal or unresponsive patient. The ED ROS was considered positive if symptoms met the CDC case definition of COVID-19.
ED attending physician PA
After the initial ROS (but prior to the COVID-19 test result being available), the attending physician would classify the patient as high, moderate, low or no probability of COVID-19. To ensure standardisation, the definition of each probability was presented in the EMR, and guidance was provided via meetings and emails. High probability patients were those with one or more of the following: known exposure to a confirmed COVID-19 case, complaints of fever and dry cough, ground glass opacities or multifocal organising pneumonia on chest imaging, low/normal white cell count and low procalcitonin. Low probability patients were those with no known exposure to a confirmed case, no significant findings on chest radiograph, high white cell count, high procalcitonin or an alternative diagnosis. Moderate probability patients had signs and symptoms that were not clearly high or low probability. For study analyses, probability was dichotomised into low/no probability and moderate/high probability.
Statistical methods
Descriptive statistics were calculated, including means and SDs for continuous variables, and frequencies and percentages for categorical variables. The frequency and percentage of each symptom documented by the NTS and ED ROS were calculated and agreement between these two methods was estimated for each symptom using simple unweighted kappa coefficients and 95% CIs14 (online supplemental appendix 1).
Supplemental material
Diagnostic characteristics including sensitivity, specificity, PPV and NPV, with corresponding 95% CIs, were calculated for attending physician PA, and if done, the NTS and ED ROS, using the results of the COVID-19 test as the reference standard. McNemar’s tests compared sensitivity and specificity values between screening methods.
Multivariable logistic regression estimated the odds of a positive COVID-19 test using the binary NTS, ED ROS and attending physician PA. Patients with unconfirmed negative PCR tests were considered negative for most analyses. A sensitivity analysis was performed where this group of patients was considered COVID-19 positive, the results of which can be found in the online supplemental appendix 1.
To assess for common groups of symptoms, a principal component analysis (PCA) using varimax rotation of symptoms was performed. Factors with an eigenvalue ≥1 were retained, which led to symptoms categorised into five groups: factor 1 (fever, chills, fatigue, sore throat, rhinorrhoea and cough), factor 2 (abdominal pain, nausea, emesis and diarrhoea), factor 3 (SOB, chest pain), factor 4 (anosmia/dysgeusia, headache) or factor 5 (anorexia, myalgias). Adjusting for age, sex and race/ethnicity, these symptom groups were included in a logistic regression model to investigate their association with a positive COVID-19 test.
Patient and public involvement
Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our research.
Results
Patient demographics
A total of 748 patients were eligible comprising 781 encounters. As we only included each patient’s first visit, the final sample size was 748 visits. The mean age was 57.5 years (SD 18.3 years), and 425 patients (56.8%) were male (table 1). Three hundred and twenty-one patients (42.9%) were black and non-Hispanic, 214 (28.6%) were white and non-Hispanic, 155 (20.7%) were Hispanic and 58 (7.8%) identified as a different racial or ethnic category or did not provide race/ethnicity information. Twenty-six per cent of patients had a primary language other than English.
Patients admitted to Boston Medical Center through the ED and screened for COVID-19 from 27 April 2020 to 17 May 2020, n=748
COVID-19 status
There were 159 (21.3%) patients with a positive COVID-19 RT-PCR and 482 (64.4%) with negative tests (table 1). There were 52 (7.0%) patients with an unconfirmed negative test who were considered negative for all analyses except for one sensitivity analysis. Fifty-five (7.3%) patients were excluded from further analysis due to missing or unknown COVID-19 test results (figure 2).
COVID-19 test results of study population.
ED attending physician PA
Seventy-four (9.9%) patients were assigned a high probability for COVID-19 by an attending physician, 131 (17.5%) a moderate probability and 451 (60.3%) no/low probability. There was no PA for 92 (12.3%) patients.
Comparison of NTS, ED ROS and attending physician PA
Among the 531 (70.9%) patients who had all three screening assessments (table 2A), the attending physician PA had the highest sensitivity (0.62, 95% CI 0.53 to 0.71) and specificity (0.76, 95% CI 0.72 to 0.80) (table 2B). The NTS had a sensitivity of 0.46 (95% CI 0.37 to 0.56) and specificity of 0.71 (95% CI 0.66 to 0.75). The ED ROS had a sensitivity of 0.53 (95% CI 0.43 to 0.62) and a specificity of 0.62 (95% CI 0.58 to 0.67). The attending physician PA was more sensitive than the NTS (0.62 vs 0.46, p=0.0115) and more specific than the ED ROS (0.76 vs 0.62, p<0.0001). The NTS was also significantly more specific than the ED ROS (0.71 compared with 0.62, p=0.002). All three methods had a low PPV ranging from 0.26 (95% CI 0.21 to 0.33) for the ED ROS to 0.40 (95% CI 0.33 to 0.47) for the attending physician PA. All three methods had a similar NPV: 0.84 (95% CI 0.80 to 0.87) for the NTS, 0.84 (95% CI 0.79 to 0.88) for the ED ROS and 0.89 (95% CI 0.85 to 0.92) for the attending physician PA. A sensitivity analysis was performed where patients with a moderate to high probability on the attending physician PA and an unconfirmed negative COVID-19 result were considered positive. In this analysis, the sensitivity and specificity of the attending physician PA were higher, and the PPV for all three methods increased slightly and the NPV for the NTS and ED ROS decreased slightly since the calculated prevalence of COVID-19 increased (online supplemental appendix 1).
(A) Distribution of patient results using each of the three screening methods, n=531. (B) Comparison of performance of three different ED screening methods, n=531
Multivariable regression analysis
A positive COVID-19 screening result obtained using any of the three methods was associated with increased odds of being COVID-19 positive in regression analyses adjusted for age, sex and race/ethnicity (table 3). The attending physician PA was associated with the highest odds of COVID-19 positivity (adjusted OR (aOR) 4.61, 95% CI 3.01 to 7.06). The NTS was also associated with increased odds of having COVID-19 (aOR 2.79, 95% CI 1.84 to 4.22) as was the ED ROS (aOR 2.13, 95% CI 1.45 to 3.12).
Multivariable associations between positive result on COVID-19 screening tool and positive COVID-19 RT-PCR test
Principal component analysis
The 323 patients with a response recorded for every symptom were included in the PCA (table 4). Only factor 1 (fever, chills, fatigue, sore throat, rhinorrhoea and cough) was associated with increased odds of testing positive for COVID-19 in unadjusted (aOR 2.94, 95% CI 1.64 to 5.27) and adjusted (aOR 2.94, 95% CI 1.60 to 5.40) analyses.
Logistic regression models (univariate and multivariable) for the outcome of having a positive COVID-19 result based on groups of symptoms classified from a PCA among 323 patients
Discussion
Three methods—an NTS, an ED ROS and an attending physician PA—were compared with regard to their ability to predict the likelihood that a patient presenting to the ED had COVID-19. Symptom screeners were necessary during the first COVID-19 surge in the spring of 2020 because testing capacity at that time was limited and we were trying to understand the presenting symptoms of a novel virus as well as cohort patients appropriately.
All three methods assessed in this study had a low sensitivity (0.46–0.62). Although symptoms such as fever, chills, fatigue, sore throat, rhinorrhoea and cough were associated with higher odds of having COVID-19, many patients with COVID-19 present with no or non-specific symptoms and still warrant testing, particularly when community prevalence of COVID-19 is high. As the pandemic has progressed, the turnaround time for COVID-19 results has decreased.15 Given the higher prevalence of asymptomatic infections with subsequent variants (eg, 2.4% with the Beta and Delta variants vs 16.0% with the Omicron variant), widespread testing remains crucial to mitigate transmission in high-risk settings.16
All three methods had a high NPV, with the attending physician PA having the highest NPV at 89%. This suggests that such screening methods may have a role to play when community transmission is lower and could be helpful in settings where there is limited PCR testing capacity.
A few factors may explain the greater sensitivity and specificity of the ED attending physician PA. First, attending physicians have more comprehensive experience with clinical decision-making than the residents, nurses and physician assistants who are completing the other two methods. Second, the attending physicians sometimes had more objective data, such as laboratory test and imaging results, when making their assessments, which is useful when predicting a patient’s likelihood of having COVID-19.17
This study elucidates challenges inherent to disease screening in the ED generally and can be used to improve screening for other diseases. During the first COVID-19 surge, the high patient volume made it challenging for personnel to complete their clinical documentation thoroughly. Data quality could be improved by using patient-completed assessments, which have been effective in collecting accurate medication histories in the ED and assessing risk of deep vein thrombosis by Caprini risk score.18 19 Smartphone-based applications have also been developed to allow patients to self-report COVID-19 symptoms and test results at home and have been used to assess factors associated with needing respiratory support.20 In the same way, patient-completed screening methods may be a more accurate, faster way to collect information to triage ED patients.
This study has several limitations. It included only a single centre, so the results are most generalisable to other large urban hospitals. The study period included a time of high prevalence of disease and hospitalisation and may not be as relevant to times of lower prevalence or severity of cases. There was likely variability in how nurses, physician assistants, residents and attending physicians completed each of the assessments, particularly the ROS. However, this reflects the real-life experience of using these methods, and highlights opportunities to make them more user-friendly going forward. Patients may not have reported all their symptoms, particularly if they were not interviewed in their primary language or if their provider had limited time to speak with them. The COVID-19 diagnosis was made using the RT-PCR test, which may have had false-negative or false-positive results.21 Finally, the reference standard was based in part on the PA, which introduces the potential for incorporation bias.
In sum, we found that the three screening methods had low sensitivity and specificity during the first COVID-19 surge in the spring of 2020. These findings suggest that in high transmission areas, hospitals do not rely on symptoms or PA in determining infection status but continue to use widespread testing. As the pandemic evolves, diagnostic testing efforts should supersede the use of clinical screening methods.
Data availability statement
No data are available.
Ethics statements
Patient consent for publication
Ethics approval
This retrospective cohort study was approved by the BMC Institutional Review Board (H-40417). The study is minimal risk and consent requirements were waived.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Handling editor Ellen J Weber
Twitter @madelinediloren
Contributors MAD, NSH, RRI, RGM and EMS-P designed the study. MAD, JND and EMS-P designed the data collection tools. MAD and JND monitored the data collection. MRD collected the data. JND and KPN wrote the statistical analysis plan and performed the statistical analysis. MAD, MRD and EMS-P drafted the paper. MAD and EMS-P are the guarantors for the study. All authors revised the paper.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests MAD owns stock in AbbVie, Agilent Technologies, Amgen, BioMarin Pharmaceuticals, Bristol Myers Squibb, Cardinal Health, CVS Health, Gilead Sciences, GlaxoSmithKline, Hologic, Jazz Pharmaceuticals, Labcorp, Merck & Co, Quest Diagnostics, Vertex Pharmaceuticals and West Pharmaceuticals. While all these companies are currently or may in the future do research related to COVID-19, none of these companies make products that were evaluated in this study.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.