Accuracy of Clinician Practice Compared With Three Head Injury Decision Rules in Children: A Prospective Cohort Study

doi:10.1016/j.annemergmed.2018.01.015

Annals of Emergency Medicine

Volume 71, Issue 6, June 2018, Pages 703-710

https://doi.org/10.1016/j.annemergmed.2018.01.015 Get rights and content

Study objective

Three clinical decision rules for head injuries in children (Pediatric Emergency Care Applied Research Network [PECARN], Canadian Assessment of Tomography for Childhood Head Injury [CATCH], and Children’s Head Injury Algorithm for the Prediction of Important Clinical Events [CHALICE]) have been shown to have high performance accuracy. The utility of any of these in a particular setting depends on preexisting clinician accuracy. We therefore assess the accuracy of clinician practice in detecting clinically important traumatic brain injury.

Methods

This was a planned secondary analysis of a prospective observational study of children younger than 18 years with head injuries at 10 Australian and New Zealand centers. In a cohort of children with mild head injuries (Glasgow Coma Scale score 13 to 15, presenting in <24 hours) we assessed physician accuracy (computed tomography [CT] obtained in emergency departments [EDs]) for the standardized outcome of clinically important traumatic brain injury and compared this with the accuracy of PECARN, CATCH, and CHALICE.

Results

Of 20,137 children, 18,913 had a mild head injury. Of these patients, 1,579 (8.3%) received a CT scan during the ED visit, 160 (0.8%) had clinically important traumatic brain injury, and 24 (0.1%) underwent neurosurgery. Clinician identification of clinically important traumatic brain injury based on CT performed had a sensitivity of 158 of 160, or 98.8% (95% confidence interval [CI] 95.6% to 99.8%) and a specificity of 17,332 of 18,753, or 92.4% (95% CI 92.0% to 92.8%). Sensitivity of PECARN for children younger than 2 years was 42 of 42 (100.0%; 95% CI 91.6% to 100.0%), and for those 2 years and older, it was 117 of 118 (99.2%; 95% CI 95.4% to 100.0%); for CATCH (high/medium risk), it was 147 of 160 (91.9%; 95% CI 86.5% to 95.6%); and for CHALICE, 148 of 160 (92.5%; 95% CI 87.3% to 96.1%).

Conclusion

In a setting with high clinician accuracy and a low CT rate, PECARN, CATCH, or CHALICE clinical decision rules have limited potential to increase the accuracy of detecting clinically important traumatic brain injury and may increase the CT rate.

Introduction

A number of clinical decision rules have been designed to assist clinicians in determining which children with head injuries are at higher or lower risk of an intracranial injury, and should therefore undergo computed tomography (CT) or do not require neuroimaging.1, 2, 3, 4, 5, 6 Some studies have assessed and compared the accuracy of different pediatric head injury clinical decision rules in prospective data sets.3, 7, 8, 9, 10 However, in addition to comparative prima facie accuracy, several other elements are important when one assesses which rule, if any, should be implemented in a particular setting. These may include baseline CT rate11, 12, 13 and effects of rule implementation,14, 15 the medicolegal climate, shared decisionmaking with families,¹⁶ availability of CT imaging and neurosurgical support, possibility to observe disease progression or recovery,17, 18 and factors such as preexisting clinician accuracy without the clinical decision rules.8, 19 Before the derivation of head injury clinical decision rules, physician accuracy was reported to be low,5, 20 instigating both the need for clinical decision rules and acceptance of their use.

Editor’s Capsule Summary

What is already known on this topic

Clinical decision rules are widely advocated to assist computed tomography (CT) use after pediatric head injury.

What question this study addressed

How do 3 clinical decision rules compare with unstructured clinical judgment in settings of nationalized health care?

What this study adds to our knowledge

In this prospective multicenter study of 18,913 children with mild head injury, clinical judgment demonstrated sensitivity similar to that of any of the 3 decision rules, as well as higher specificity than any of them.

How this is relevant to clinical practice

In these nationalized health care settings, clinical decision rules for pediatric head injury did not improve on clinical judgment and would likely increase CT use.

We recently completed a study on the comparative accuracy of 3 high-quality clinical decision rules,³ the prediction rule for the identification of children at very low risk of clinically important traumatic brain injury developed by the Pediatric Emergency Care Applied Research Network (PECARN),⁴ the Canadian Assessment of Tomography for Childhood Head Injury (CATCH) rule,⁵ and the Children’s Head Injury Algorithm for the Prediction of Important Clinical Events (CHALICE).⁶ In a comparison cohort of 18,913 head-injured children with Glasgow Coma Scale (GCS) score 13 to 15, the point sensitivities of the rules in our external validation cohort were high.

In settings with high baseline CT rates and high variability of CT rates, as has been reported in the United States and Canada,11, 12 clinical decision rules may assist in safely reducing CT rates.15, 16 In settings with low CT rates, such as that reported in Australia,3, 13, 21 the introduction of clinical decision rules has the potential to increase neuroimaging, with unclear benefit in detecting intracranial injuries.3, 19 Key to understanding the benefits of clinical decision rules in a particular setting will be to know clinician accuracy without formal use of clinical decision rules.²²

Using a large cohort of mildly head-injured children (GCS score 13 to 15) in a setting with low CT imaging rates, we set out to assess clinician accuracy (sensitivity and specificity) according to whether a CT was performed during the initial emergency department (ED) visit. We then compared clinician accuracy with the accuracy of PECARN, CATCH, and CHALICE, using a single outcome measure across all rules, clinically important traumatic brain injury.

Section snippets

Study Design and Setting

This was a planned substudy of a prospective multicenter observational study that enrolled children younger than 18 years with head injury of any severity who presented to 10 pediatric EDs in Australia and New Zealand between April 2011 and November 2014.3, 23 All EDs are members of the Paediatric Research in Emergency Departments International Collaborative research network.²⁴

The study sites had a census ranging from 19,000 to 78,000 children treated annually. Seven of the 10 EDs were regional

Characteristics of Study Subjects

A total of 29,433 patients presented to the ED with head injury of any severity, of whom 5,203 were missed. Of 20,137 evaluable patients, 18,913 had a GCS score of 13 to 15 and presented within 24 hours (Figure). Mean age in this cohort was 5.7 years (SD 4.6 years). Most injuries were due to falls (70.5%). Overall, 1,691 patients (8.9%) received a CT scan at any time in relation to the head injury, 1,579 (8.3%) received a CT scan during their initial ED visit, and 24 (0.1%) underwent

Limitations

This study was conducted at mostly tertiary Australian and New Zealand pediatric centers with pediatric emergency physicians on staff and so may not be representative of care at general and mixed EDs.

Although we do not know whether clinicians in this study followed one of the known clinical decision rules or incorporated elements of them, in a survey before conducting this study we found that none of the known head injury clinical decision rules predominated in senior clinician practice or

Discussion

In this multicenter study, clinician accuracy in detecting clinically important traumatic brain injury was very high, whether neuroimaging alone or the combination of neuroimaging and observation was considered as the criterion for clinician accuracy. When clinician accuracy (sensitivity 99%) was compared with 3 high-quality clinical decision rules, PECARN had similarly high point sensitivities (99% to 100%); CATCH and CHALICE had lower point sensitivities but 95% CIs overlapped across all

References (29)

F.E. Babl et al.
Paediatric Research in Emergency Departments International Collaborative (PREDICT). Accuracy of PECARN, CATCH and CHALICE head injury decision rules in children. A prospective cohort study
Lancet
(2017)
N. Kuppermann et al.
Identification of children at very low risk of clinically-important brain injuries after head trauma: a prospective cohort study
Lancet
(2009)
R.M. Stanley et al.
Emergency department practice variation in computed tomography use for children with minor blunt head trauma
J Pediatr
(2014)
J.F. Holmes et al.
Do children with blunt head trauma and normal cranial computed tomography scan results require hospitalization for neurologic observation?
Ann Emerg Med
(2011)
D. Schonfeld et al.
Effect of the duration of emergency department observation on computed tomography use in children with minor blunt head trauma
Ann Emerg Med
(2013)
P.M. Bossuyt et al.
Toward complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Acad Radiol
(2003)
S.M. Green et al.
Methodologic standards for interpreting clinical decision rules in emergency medicine: 2014 update
Ann Emerg Med
(2014)
P.A. Harris et al.
Research Electronic Data Capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support
J Biomed Inform
(2009)
I.G. Stiell et al.
The Canadian CT Head Rule study for patients with minor head injury: rationale, objectives, and methodology for phase I (derivation)
Ann Emerg Med
(2001)
A. Pickering et al.
Clinical decision rules for children with minor head injury: a systematic review
Arch Dis Child
(2011)

M.D. Lyttle et al.

Comparing CATCH, CHALICE and PECARN clinical decision rules for paediatric head injuries

Emerg Med J

(2012)

M.H. Osmond et al.

CATCH: a clinical decision rule for the use of computed tomography in children with minor head injury

CMAJ

(2010)

J. Dunning et al.

Derivation of the children's head injury algorithm for the prediction of important clinical events decision rule for head injury in children

Arch Dis Child

(2006)

M.D. Lyttle et al.

Applicability of the CATCH, CHALICE and PECARN paediatric head injury clinical decision rules: pilot data from a single Australian centre

Emerg Med J

(2013)

Cited by (32)

ACR Appropriateness Criteria® Head Trauma: 2021 Update
2021, Journal of the American College of Radiology
Citation Excerpt :
One large retrospective study of 4,554 mild head trauma encounters between 2009 and 2014 found a paradoxical increase in CT utilization (81.6%–87.6%) and decrease in CT yield for intracranial findings (12.2%–9.6%) after guideline implementation in 2011 [21]. Comparison studies from Australia and New Zealand determined clinical decision rules to be less specific than usual care by clinicians, contrasting with studies from the United States and reflecting differences in baseline scan rates [22,23]. Ongoing efforts to improve the specificity and positive predictive value of clinical decision rules include the application of machine learning techniques (eg, artificial neural networks and optimal classification trees), which come at the cost of complexity [24,25].
Head trauma (ie, head injury) is a significant public health concern and is a leading cause of morbidity and mortality in children and young adults. Neuroimaging plays an important role in the management of head and brain injury, which can be separated into acute (0–7 days), subacute (<3 months), then chronic (>3 months) phases. Over 75% of acute head trauma is classified as mild, of which over 75% have a normal Glasgow Coma Scale score of 15, therefore clinical practice guidelines universally recommend selective CT scanning in this patient population, which is often based on clinical decision rules. While CT is considered the first-line imaging modality for suspected intracranial injury, MRI is useful when there are persistent neurologic deficits that remain unexplained after CT, especially in the subacute or chronic phase. Regardless of time frame, head trauma with suspected vascular injury or suspected cerebrospinal fluid leak should also be evaluated with CT angiography or thin-section CT imaging of the skull base, respectively.
The American College of Radiology Appropriateness Criteria are evidence-based guidelines for specific clinical conditions that are reviewed annually by a multidisciplinary expert panel. The guideline development and revision include an extensive analysis of current medical literature from peer reviewed journals and the application of well-established methodologies (RAND/UCLA Appropriateness Method and Grading of Recommendations Assessment, Development, and Evaluation or GRADE) to rate the appropriateness of imaging and treatment procedures for specific clinical scenarios. In those instances where evidence is lacking or equivocal, expert opinion may supplement the available evidence to recommend imaging or treatment.
Validation of the PECARN head trauma prediction rules in Japan: A multicenter prospective study
2020, American Journal of Emergency Medicine
Citation Excerpt :
As the CT rate was only 7.0% in our study, the PECARN rules may not have reduced unnecessary CT scans in the participating centers. As Babl et al. reported, in a setting with a low CT rate, head trauma prediction rules have limited potential to increase diagnostic accuracy and may increase the CT rate [16]. However, there is a wide range of CT rates depending on the treating physicians or facilities in Japan.
Head trauma in children is one of the most common causes for emergency department visits. Although most trauma cases are minor, identifying those patients who have clinically important traumatic brain injury (ciTBI) is challenging. The Pediatric Emergency Care Applied Research Network (PECARN) head trauma prediction rules identifying children who do not require cranial computed tomography (CT) were validated and are used all over the world. However, these rules have not been validated with large cohort multicenter studies in Asia.
To investigate whether the PECARN rules can be safely applied to Japanese children.
We conducted a multicenter, prospective, observational cohort study. We included children younger than 16 with minor head trauma (Glasgow Coma Scale ≥14) who presented to the six participating centers within 24 h of their injuries between June 2016 and September 2017.
The primary analysis was set to calculate the negative predictive value of the patients with very low risk by the PECARN rules, compared with a preset threshold of 99.85%.
We included 6585 children of which 463 (7.0%) had head CT scans performed and 23 (0.35%) had ciTBI. There were two patients with ciTBI who were classified as very low risk. The negative predictive value, calculated as 99.96% (95%CI: 99.86–100.00; P = .019), was significantly superior compared with the preset threshold of 99.85%.
The PECARN head trauma prediction rules seemed to be safely applicable to Japanese children. Further studies are needed to determine safety in hospitals where physicians do not have expertise in managing children.
Systematic review and validation of diagnostic prediction models in patients suspected of meningitis
2020, Journal of Infection
Citation Excerpt :
They found that in their center accuracy of physicians in terms of sensitivity and specificity was high.44 Therefore, the decision rules had limited potential to increase the diagnostic accuracy and could even increase the rate of unnecessary CT-scans.44 The results of both studies show the importance of prospective comparison of physicians’ performance with and without prediction models, for instance in a cluster randomized design.
Diagnostic prediction models have been developed to assess the likelihood of bacterial meningitis (BM) in patients presented with suspected central nervous system (CNS) infection. External validation in patients suspected of meningitis is essential to determine the diagnostic accuracy of these models.
We prospectively included patients who underwent a lumbar puncture for suspected CNS infection. After a systematic review of the literature, we applied identified models for BM to our cohort. We calculated sensitivity, specificity, predictive values, area under the curve (AUC) and, if possible, we evaluated the calibration of the models.
From 2012-2015 we included 363 episodes. In 89 (24%) episodes, the patient received a final diagnosis of a CNS infection, of whom 27 had BM. Seventeen prediction models for BM were identified. Sensitivity of these models ranged from 37% to 100%. Specificity of these models ranged from 44% to 99%. The cerebrospinal fluid model of Oostenbrink reached the highest AUC of 0.95 (95% CI 0.91–0.997). Calibration showed over- or underestimation in all models.
None of the existing models performed well enough to recommend as routine use in individual patient management. Future research should focus on differences between diagnostic accuracy of the prediction models and physician's therapeutic decisions.
Sports Medicine Update: Concussion
2020, Emergency Medicine Clinics of North America
Citation Excerpt :
There are several low-risk criteria to guide the decision to perform CT imaging in pediatric patients, including Pediatric Emergency Care Applied Research Network (PECARN), Children's Head Injury Algorithm for the Prediction of Important Clinical Events (CHALICE), and Canadian Assessment of Tomography for Childhood Head injury (CATCH).65–67 Although all 3 rules have excellent negative predictive values, PECARN is thought to have greater sensitivity and may be the preferred tool.68,69 As with adults, a systematic approach guided by the SCAT5 (patients aged 13 years and older) or the Child SCAT5 (patients aged 12 years and younger) to the history and physical examination helps to identify concussed patients.21
Implementation of a Clinical Decision Support System for Children With Minor Blunt Head Trauma Who Are at Nonnegligible Risk for Traumatic Brain Injuries
2019, Annals of Emergency Medicine
Citation Excerpt :
Decreased CT use was not observed, however, at the control sites, which is consistent with recent data on temporal trends for CT use in this population.7 There was no associated increase in CT use or hospitalization for any group of children at nonnegligible risk of clinically important traumatic brain injuries, as might be feared with commensurate implementation of a low-risk prediction rule.20,31,32 The lack of significant effects of our intervention on ED length of stay was also reassuring, suggesting no significant influence of clinical decision support on duration of ED evaluations.
To determine the effect of providing risk estimates of clinically important traumatic brain injuries and management recommendations on emergency department (ED) outcomes for children with isolated intermediate Pediatric Emergency Care Applied Research Network clinically important traumatic brain injury risk factors.
This was a secondary analysis of a nonrandomized clinical trial with concurrent controls, conducted at 5 pediatric and 8 general EDs between November 2011 and June 2014, enrolling patients younger than 18 years who had minor blunt head trauma. After a baseline period, intervention sites received electronic clinical decision support providing patient-level clinically important traumatic brain injury risk estimates and management recommendations. The following primary outcomes in patients with one intermediate Pediatric Emergency Care Applied Research Network risk factor were compared before and after clinical decision support: proportion of ED computed tomography (CT) scans, adjusted for age, time trend, and site; and prevalence of clinically important traumatic brain injuries.
The risk of clinically important traumatic brain injuries was known for 3,859 children with isolated findings (1,711 at intervention sites before clinical decision support, 1,702 at intervention sites after clinical decision support, and 446 at control sites). In this group, pooled CT proportion decreased from 24.2% to 21.6% after clinical decision support (odds ratio 0.86; 95% confidence interval 0.73 to 1.01). Decreases in CT use were noted across intervention EDs, but not in controls. The pooled adjusted odds ratio for CT use after clinical decision support was 0.73 (95% confidence interval 0.60 to 0.88). Among the entire cohort, clinically important traumatic brain injury was diagnosed at the index ED visit for 37 of 37 (100%) patients before clinical decision support and 32 of 33 patients (97.0%) after clinical decision support.
Providing specific risks of clinically important traumatic brain injury through electronic clinical decision support was associated with a modest and safe decrease in ED CT use for children at nonnegligible risk of clinically important traumatic brain injuries.
Clinical Judgment and the Pediatric Emergency Care Applied Research Network Head Trauma Prediction Rules
2018, Annals of Emergency Medicine

View all citing articles on Scopus

: Please see page 704 for the Editor’s Capsule Summary of this article.

: Supervising editor: Steven M. Green, MD

: Author contributions: FEB conceived the study, obtained grant funding, wrote the initial draft of the article, and provided overall supervision. All authors designed the study, approved publication, and agreed to be accountable for all aspects of the work. FEB, EO, SRD, MLB, NP, AK, S. Dalton, JAC, YG, JF, JN, SH, CM, LC, SB, and MDL interpreted the data. EO, SRD, MLB, NP, AK, S. Dalton, JAC, YG, JF, JN, LC, SB, and MDL obtained the data, provided supervision, and drafted or revised the paper critically. S. Donath supervised the analysis of the data, contributed to the interpretation of the data, and revised the article critically. FEB takes responsibility for the paper as a whole.

: All authors attest to meeting the four ICMJE.org authorship criteria: (1) Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND (2) Drafting the work or revising it critically for important intellectual content; AND (3) Final approval of the version to be published; AND (4) Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

: Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article as per ICMJE conflict of interest guidelines (see www.icmje.org). The authors have stated that no such relationships exist. The study was funded by grants from the National Health and Medical Research Council (project grant GNT1046727, Centre of Research Excellence for Paediatric Emergency Medicine GNT1058560), Canberra, Australia; the Murdoch Children’s Research Institute, Melbourne, Australia; the Emergency Medicine Foundation (EMPJ-11162), Brisbane, Australia; Perpetual Philanthropic Services (2012/1140), Australia; Auckland Medical Research Foundation (No. 3112011) and the A + Trust (Auckland District Health Board), Auckland, New Zealand; WA Health Targeted Research Funds 2013, Perth, Australia; and the Townsville Hospital and Health Service Private Practice Research and Education Trust Fund, Townsville, Australia. The study was also supported by the Victorian Government’s Infrastructure Support Program, Melbourne, Australia. Dr. Babl’s time was partly funded by a grant from the Royal Children’s Hospital Foundation, Melbourne, Australia; a Melbourne Children’s Clinician Scientist Fellowship, Melbourne, Australia; and an NHMRC Practitioner Fellowship, Canberra, Australia. Dr. Dalziel’s time was partly funded by the Health Research Council of New Zealand (HRC13/556).

: Trial registration number: ACTRN12614000463673

: Readers: click on the link to go directly to a survey in which you can provide feedback to Annals on this particular article.

: A podcast for this article is available at www.annemergmed.com.

View full text

Pediatrics/original researchAccuracy of Clinician Practice Compared With Three Head Injury Decision Rules in Children: A Prospective Cohort Study

Study objective

Methods

Results

Conclusion

Introduction

Editor’s Capsule Summary

Section snippets

Study Design and Setting

Characteristics of Study Subjects

Limitations

Discussion

Lancet

Lancet

J Pediatr

Ann Emerg Med

Ann Emerg Med

Acad Radiol

Ann Emerg Med

J Biomed Inform

Ann Emerg Med

Clinical decision rules for children with minor head injury: a systematic review

Arch Dis Child

Comparing CATCH, CHALICE and PECARN clinical decision rules for paediatric head injuries

Emerg Med J

CATCH: a clinical decision rule for the use of computed tomography in children with minor head injury

CMAJ

Derivation of the children's head injury algorithm for the prediction of important clinical events decision rule for head injury in children

Arch Dis Child

Applicability of the CATCH, CHALICE and PECARN paediatric head injury clinical decision rules: pilot data from a single Australian centre

Emerg Med J

Pediatrics/original research
Accuracy of Clinician Practice Compared With Three Head Injury Decision Rules in Children: A Prospective Cohort Study