Objective To measure the performance characteristics of an immunochromatographic rapid antigen test for respiratory syncytial virus (RSV) and determine how its interpretation should be contextualised in patients presenting to the emergency department (ED) with bronchiolitis.
Design Diagnostic accuracy study of a rapid RSV test.
Setting County hospital ED.
Intervention We took paired nasal samples from consecutively enrolled infants with bronchiolitis and tested them with a rapid immunochromatographic antigen test and reverse transcriptase PCR gold standard.
Outcome measures Sensitivity, specificity, the effect of point prevalence, clinical findings and overall context on predictive values. We used these to construct a graphical contextual model to show how the results of RSV antigen tests from infants presenting within 24 h should influence interpretation of subsequent antigen tests.
Results We analysed 607 patients. The sensitivity and specificity for immunochromatographic testing was 79.4% (95% CI 73.9% to 84.2%) and 67.1% (95% CI 61.9% to 72%) respectively. We found little evidence of spectrum bias. In our contextual model the best predictor of a positive RT-PCR test was a positive antigen test OR 5.47 (95% CI 3.65 to 8.18) and the number of other infants having positive tests within 24 h OR 1.48 (95% CI 1.26 to 1.72) per infant. Increasing numbers presenting to the ED with bronchiolitis in a given day increases the probability of RSV infection.
Conclusions The RSV antigen test we examined had modest performance characteristics. The results of the antigen test should be interpreted in the context of the results of previous tests.
- Respiratory, Pneumonia/Infections
- Infectious Diseases
Statistics from Altmetric.com
Respiratory syncytial virus (RSV) infection is ubiquitous. Most children have serological evidence of RSV infection by 2 years of age.1 ,2 RSV is the commonest cause of bronchiolitis,3 ,4 and one of the commonest causes of infant hospitalisation.5 ,6
Diagnostic confirmation of RSV infection greatly decreases the risk of coexisting bacterial illness, particularly in the presence of bronchiolitis.7–11 Clinicians can often safely limit or omit testing and treating for bacterial infection if RSV is confirmed. Similarly, although any infection can induce central apnoea, RSV is the commonest cause.12 Consequently, RSV testing is common in emergency departments (ED). As RSV-specific antiviral agents become available, emergency physicians will come under pressure to prescribe them, and therefore to test for RSV.
Clinical laboratories typically rely on easily performed rapid immunochromatographic antigen (RA) tests to detect RSV. The results are available in approximately 20 min and are reported as positive, negative, or rarely equivocal. To ascertain if this apparent simplicity of interpretation is justified we asked three questions.
First, what is the independently measured sensitivity and specificity of an RA RSV test using PCR (RT-PCR) as the reference standard? Independent assessment rather than relying on manufacturer-supplied data is important as others have failed to reproduce manufacturer-claimed performance in other tests.13–15
Second, how does this diagnostic performance affect a test result given local RSV prevalence? For clinicians, the pertinent question is, if the patient tests positive for RSV what is the probability that this patient in fact has RSV? The probability that a positive test is truly positive is related to how prevalent RSV is in infants at the time of testing, and community virological prevalence is associated with clinical disease.16
Third, should clinical context modify interpretation of rapid RSV test results? We defined clinical context as the physical examination findings and severity of illness in the patient being tested.
Overall context incorporates both local prevalence and clinical findings. A very large study showed that modifying the Centor score based on local bacteriological incidence data in the evaluation of streptococcal pharyngitis would have substantially improved diagnostic accuracy.17 In our study, overall context incorporated clinical findings with the results of antigen tests performed on other infants who presented with bronchiolitis within 24 h of the infant being tested. Our approach addresses both the recognised seasonality of RSV, and the practical problem of never being quite sure where one is in a given season until after the season has ended.
We conducted a prospective diagnostic accuracy study of a rapid immunochromatographic RSV antigen test from November 2006 to April 2009. We defined RT-PCR as the gold standard. Our outcomes were sensitivity, specificity, the effect of point prevalence, clinical findings and overall context on predictive values for the rapid immunochromatographic antigen test.
Emergency department in a county teaching hospital.
We included patients who had a clinical diagnosis of bronchiolitis, and in whom the clinician ordered a rapid RSV test. We defined bronchiolitis operationally as evidence (eg, wheezing, chest wall retractions) of lower airway obstruction following a period of upper respiratory tract symptoms up to the 18 month of life. Research assistants identified potential subjects by real-time triage note review and observation of waiting patients. Eligibility was determined by mid-level providers, faculty and selected resident physicians. We obtained written, informed consent from the parent/guardian.
Patients were excluded if the clinician did not intend to order a rapid RSV test, consent was refused, RSV testing was obtained prior to screening, or if an alternative transport or collection system to that specified was used.
Our primary outcome measures were: RSV antigen test results, positive or negative (we repeated equivocal tests), and RT-PCR RSV results positive or negative. These were used to calculate antigen test characteristics. Secondary outcomes included the presence of an additional or alternative aetiology by RT-PCR, specifically human metapneumovirus (HMV) or influenza (INF) A or B.
We collected samples for antigen testing using saline anterior nasal aspirates as described by Hall.18 These specimens were immediately sent to our clinical laboratory for testing. We obtained paired samples for reverse transcriptase PCR (RT-PCR) testing with nasal mid-turbinate nylon-flocked swabs placed in universal transport medium-room temperature (UTM-RT), (Copan Diagnostics, Murrieta, California). The UTM-RT-swab specimens were frozen to −20°C prior to being shipped on dry ice for batch testing. This collection and transport system has been described in detail elsewhere.19 The order of swab or wash was not specified. Each collection system was performed on opposite nostrils.
We performed immunochromatographic testing using Directigen RSV, (Becton Dickinson, Sparks, Maryland) in accordance with the manufacturer's instructions in the hospital clinical laboratory by certified laboratory technicians.20 This test uses antibodies against the fusion and nucleocapsid proteins. RT-PCR testing was performed in batches at a remote research laboratory. The laboratory performing the RT-PCR was blinded to the antigen test results. Blinding of the clinical laboratory was straightforward because the PCR results were not available until after the antigen test was reported. Blinding the laboratory that performed the RT-PCR was ensured by using only an alphanumeric sample code to identify samples. The file linking patient identifiers to their sample code was not accessible to either laboratory's staff. The RT-PCR assay is described in the Appendix.
We also performed RT-PCR testing for three other pathogens: HMV, INF A and INF B on all specimens. All RT-PCR tests had a minimum detection threshold of 100 genomic copies. We did this rather than attempting direct immunoflourescence (DFA) or viral culture as a ‘tiebreaker.’ DFA is a highly microscopist-dependent technique. The relative fragility of the RSV virus would result in a falsely low detection rate because viral culture was not immediately available.
Sample size calculation
Assuming half the subjects would truly have RSV, an RA test sensitivity of 90% +/− 5%, a power of 90% and a significance level of 5%, we would have required 154 patients.
Anticipating the multivariate models contemplated by our study questions we followed Long's guideline of at least 500 subjects and anticipated requiring three bronchiolitis seasons to achieve this.21
Calculation of diagnostic accuracy
We calculated sensitivity, specificity, predictive values and likelihood ratios (LR) using the diagt command in Stata.22 Spectrum bias occurs when supposedly stable test characteristics, such as sensitivity and specificity, actually differ between groups. For example; echocardiography may detect central pulmonary emboli but not smaller peripheral ones. We sought spectrum bias by repeating these calculations for the following subgroups: age <2 months, increased work of breathing, hypoxia (SaO2 <92%), and National Children's Hospital Severity of Bronchiolitis (NCH-SoB) score. The NCH-SoB score is a validated ordinal regression model which classifies bronchiolitis as mild, moderate, or severe.23 These subgroups were chosen a priori for their clinical relevance.
We addressed discordant RT-PCR and immunochromatographic RSV tests by comparing the prevalence of alternative aetiologies. The alternative aetiologies we tested for were HMV and influenza (INF) A and B. We compared the prevalence of an alternative aetiology when the immunochromatographic RSV test was positive but the RT-PCR negative, with the prevalence of an alternative aetiology when the immunochromatographic RSV test was negative but the RT-PCR was positive using Fisher's exact test. Since there are no conserved epitopes between these viruses more alternative diagnoses among infants who had negative RT-PCR tests in the face of a positive RSV antigen test would reassure us of our criterion reference's performance.
Estimating the effects of prevalence on test performance
We used the sensitivity and specificity calculated on the whole sample to recalculate and graph the positive and negative predictive values for all possible prevalence. We used LRs to construct a Fagan nomogram.24 The Fagan nomogram allows users to calculate the post-test probability that an infant has RSV given baseline prevalence and the test result.
Estimating the effect of prior test results
In practice, clinicians rarely know what local prevalence is, especially for RSV whose seasons vary by year and location. We used logistic regression to model the effect of the antigen test and prior antigen tests on RT-PCR proven RSV. The first two models modelled only additional positive or negative prior tests to allow for a clear one-dimensional depiction of the process. The third model allowed combinations of positive and negative tests within 24 h. We created a two-panel colour-coded graph to allow visualisation of these effects; one panel for the interpretation of prior results given a new positive antigen test and one panel for a new negative antigen test.
Estimating the effect of clinical characteristics on test performance
We performed univariate analysis on clinical characteristics that we thought may influence test performance. Those with a p value of <0.1 were eligible for inclusion in the model. Rather than relying on automated forward or backward stepwise regression procedures we manually selected plausible candidate variables and combination of variables. We incorporated these variables into the models described above and compared model fit and diagnostics. The institutional review board of Kern Medical Center approved this study.
We enrolled 615 patients. Although it was theoretically possible to enrol patients in summer, as a practical matter none were. Eight (1.3%) patients were excluded for lack of paired antigen/swab testing. Of the remaining 607 patients, 346 (57.0%) were male. Most had moderately severe bronchiolitis. The patients are described in table 1. Figure 1 shows the antigen and RT-PCR results in standards for the reporting of diagnostic accuracy studies (STARD) format available at http://www.stard-statement.org.
The sensitivity and specificity for antigen testing were 79.4% (95% CI 73.9% to 84.2%) and 67.1% (95% CI 61.9% to 72%) respectively. The area under the receiver operating curve (ROC) was 0.73. Predictive values and their CIs for all possible prevalence are shown in figure 2. This should not be confused with the more familiar ROC. Whereas the ROC allows the user to determine optimum sensitivity and specificity cut points for a numeric test, figure 2 shows what the predictive value is for positive and negative tests for all possible prevalence. Figure 2 also shows that the discriminating ability of the antigen test peaks when the prevalence of RSV is 53.5%.
Table 2 shows the performance of the antigen test in various subgroups. The antigen test may perform slightly better with moderately severe bronchiolitis and increased work of breathing, but overall we found scant evidence of spectrum bias.
The Fagan nomogram (figure 3) shows how the antigen test result changes the post-test probability of RSV being present. Drawing a line from the pretest probability through the positive (for a positive test) or negative (for a negative test) LRs on the central axis will show the post-test probability of RT-PCR proven RSV. When pretest probability is uncertain, local prevalence is often substituted. For illustrative purposes we have drawn the lines where the pretest probability is 50%.
Clinical factors added little to our contextual multivariate models. Figure 4 is a graphical representation of the most parsimonious model that included the infant’s own RSV antigen test result and a count of the number of other infants who had positive immunochromatographic tests that day. This model showed that the best predictor of a positive RT-PCR test was a positive antigen test OR 5.47 (95%CI 3.65 to 8.18), and the number of other infants having positive tests within 1 day OR 1.48 (95%CI 1.26 to 1.72). Figure 4 demonstrates how as increasing numbers of infants present to the ED with bronchiolitis the more likely RSV is responsible.
Figure 5 takes this further showing the marginal probabilities of a positive RSV RT-PCR test given a current positive (upper pane) or negative (lower pane) antigen test and the number of previous positive and negative antigen tests within 24 h. The user picks the pane depending on whether the current test is positive or negative and then draws lines from the number of prior negative tests (the y-axis) and from the number of prior positives (x-axis). The colour legend shows the probability of RT-PCR proven RSV. This also shows that, as large numbers of children arrive at the ED with bronchiolitis, the probability that RSV will be the cause rises even if many of the earlier children had negative antigen tests.
RSV antigen testing is used for diagnostic certainty. Such certainty is sometimes useful and can eliminate the need for more invasive tests. However, we found that antigen test interpretation is not as simple as positive or negative. The antigen test had substantially lower sensitivity and specificity than that reported by the manufacturer. We also found that RA test interpretation is best contextualised, that is interpreted in light of the presentation of other infants to the ED within 24 h. This approach is more useful to practicing clinicians than expecting them to know their community's prevalence of RSV.
The number of positive antigen tests that were negative on RT-PCR testing was striking. The antigen test kits were from several batches and were correctly performed by certified technicians. The RT-PCR detection threshold was 100 copies and carried out in a commercial laboratory. All the RT-PCR samples passed internal quality controls. Consequently, although disappointing, the results are likely correct. The resultant sensitivity and specificity results contrast with those obtained by some who have found better performances for antigen tests.15 Conversely, others have found even poorer performance for RA RSV testing than we did.25
Two processes may explain the differences between studies such as ours and those showing better test performance. First, in the early stages of test development, it is reasonable to evaluate test performance in groups of patients at very high and very low risk for the target condition (eg, a pregnancy test might be evaluated in men and women on a maternity ward. Any positives in men or negatives in the maternity ward patients allow the test to be discarded without additional expenditure of resources). Although a useful first step, this initial testing is insufficient. A diagnostic test must be able to detect the target condition where both the group with and without the condition are in other respects similar. Bronchiolitis is such a condition. The antigen test seemed calibrated to have maximal discriminatory ability when the RSV positive bronchiolitis rate approximated 50%. A second cause for these differences may be subtle differences in study design and blinding. For example, one protocol for a rapid streptococcal test required that tonsillar exudates first be swabbed with the antigen swab being investigated and then for the confirmatory swab for culture. Other differences include choices of gold standard; this is particularly important if culture is relied upon for fragile viruses or microscopist dependent techniques are used. Finally, regulatory agencies may dictate the gold standard used. This can invite designs that lead to better results than clinically oriented investigators using RT-PCR subsequently find. Although our results contrast with those of some, our results for the antigen test performance are similar to those obtained by other independent investigators of similar tests.13 ,14 ,26 ,27
Our graphical presentation of contextual diagnostic test modelling is novel and facilitates rapid interpretation. Consider the following clinical vignette.
Clinicians A and B are in different EDs. Both are evaluating a febrile 3-month old infant with bronchiolitis. Both are considering an intervention where RSV status matters, say a new RSV drug or a urinalysis. Clinician A has seen two other patients whose RSV antigen tests were negative and one whose test was positive. The current patient is RSV positive. Referring to figure 5, this patient's probability of actually having RSV is 60–70%. Clinician B has seen five patients whose RSV antigen tests were negative and two in whom it was positive. The current patient's RSV antigen test is negative. Referring to figure 5, this patient's probability of actually having RSV is 40–50%. Clinician A's management may now hinge on the relative costs of the intervention. Clinician B is little better off than if he had flipped a coin.
This also suggests a role for hospital laboratories; rather than reporting RSV antigen tests as positive or negative, the results of tests from the previous 24 h could be leveraged to provide a probability estimate of what the RT-PCR result would be.
Contextualised test interpretation by incorporating results from previous patients has been shown to improve the performance of diagnostic algorithms for meningitis and pertussis.28 ,29 Generally, these have been presented as modifications to pre-existing rules for the end user. For example, in the case of meningitis, a previously described rule was modified to include a variable ‘Were there four or more cases of viral meningitis in the preceding 10 days?’ This requires the treating doctor to know how many cases of viral meningitis were diagnosed in his department in the preceding 10 days.28 Our graphical presentation allows the clinician estimate the risk based on the data to hand and is a logical progression from prior contextual constructs.
Our study had several limitations. We did not use a third method to resolve discrepant results. Consequently, it is possible that, when discordant, RA tests were correct, and the RT-PCR results incorrect. However, the finding of significantly more alternative aetiologies when the RA test was positive but the RT-PCR negative for RSV than the converse supports our choice of gold standard. We did not test for other viruses, such as rhinoviruses or adenoviruses. This was based partly on resources and partly because rhinovirus in particular is prevalent regardless of clinical presentation. We did not test for bacterial infections. Our rate of RSV was lower than others; this likely reflects both our methodology which required diagnosis without knowledge of the antigen result and our relatively high prevalence of HMV. The sample collection methods for RA testing and RT-PCR were different, but reflect the preferred methods for each. Our results should not be extrapolated to other tests, collection and transport systems, the elderly or manifestations of RSV other than bronchiolitis. We confined our study to bronchiolitis to limit the breadth of patients our contextual model would have to account for; a broader model would have been a much more ambitious undertaking. Finally, we have not addressed the value of diagnostic certainty itself. This is a different and thorny topic. We have established only that diagnostic certainty cannot be established with a widely used antigen test, but that contextualised interpretation improves its performance.
The RA RSV antigen test we examined has modest performance characteristics and is best interpreted in the context of previous patients’ results. Increasing numbers of infants presenting to the ED with bronchiolitis in a given day increases the probability of RSV being present even if initial RA RSV tests are negative.
The authors acknowledge the input of the following: Nathan Kuppermann MD (1), Nicholas Kenyon MD (1), Valerie Aguilar BS (4), and the Emergency medicine research assistants program at Kern Medical Center. PW had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Contributors PW, Design, data management & analysis, critical contribution to manuscript. NW, data collection, data management, critical contribution to manuscript. LI, data collection, data management, CL, data collection, data management, critical contribution to manuscript. JP, data management, critical contribution to manuscript. CH, data collection, data management, critical contribution to manuscript. LS, data collection, data management, critical contribution to manuscript. TN, data collection, data management, critical contribution to manuscript. EC, data collection, data management, critical contribution to manuscript. JH, data collection, data management, critical contribution to manuscript. EM, Funding, data collection, data management, critical contribution to manuscript. MEA, data collection, data management, critical contribution to manuscript. KTI, data collection, data management, critical contribution to manuscript.
Funding This work was supported by Award Number 5K12HL108964–02 from the National Heart, Lung, and Blood Institute at the National Institutes for Health and by The Paediatric Emergency Medicine Research Foundation, Long Beach, CA. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Heart, Lung, and Blood Institute or the National Institutes of Health or The Paediatric Emergency Medicine Research Foundation.
Competing interests Neither the clinical laboratory that performed antigen testing nor the research laboratory teams was allowed a role in data analysis. Both laboratory teams were kept blinded to the other team's results. MDL provides PCR testing on a commercial basis. Copan Innovation (Murrieta, CA) provided swabs and media. Neither company was allowed a role in study design, execution or analysis.
Ethics approval Institutional review board of Kern Medical Center.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Limited de-identified data set and statistical code available from the author with some use limitations.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.