Article Text

Download PDFPDF

Side effects of decision rules, or the law of unintended consequences
  1. Ellen J Weber1,
  2. Edward W Carlton2,3
  1. 1 Emergency Medicine, University of California San Francisco, San Francisco, California, USA
  2. 2 Emergency Department, Poole Hospital NHS Foundation Trust, Poole, UK
  3. 3 School of Health and Social Care, Bournemouth University, Bournemouth, UK
  1. Correspondence to Dr Ellen J Weber, Emergency Medicine, University of California San Francisco, San Francisco CA 94143, USA; ellen.weber{at}ucsf.edu

Statistics from Altmetric.com

Over the course of the past few decades, we have seen a major swing in the value placed on physician independent decision making. The independently minded physician, practising the art of medicine, using a combination of knowledge, experience and a little gestalt, is no longer the epitome of good practice; instead, researchers derive an increasing number of decision rules, and our societies and healthcare payers formulate guidance, which the good physician is expected to know and follow. For the most part, this is a good thing. Medicine has gotten increasingly complex, the knowledge base exceedingly large and the costs exploding; the individual physician would have trouble critically appraising all the literature in their field, and determining the most clinically effective and cost-effective way to proceed.

However, as is the case with almost all medical remedies, there are side effects, even adverse reactions, to decision rules.

Emergency medicine has plenty of decision rules: rules for head injury (in adults and children), c-spine injury, pulmonary embolism (PE), ankle and knee injuries and kidney stones, not to mention countless chest pain rules. The primary goal for all of these rules is to avoid missed diagnosis. When deriving decision rules, researchers often set out to achieve one overarching goal—a sensitivity approaching 100%, often at the cost of specificity. This means that a number of patients who do not need the tests will still get them. Therefore, when applying these decision rules in everyday practice, physicians are paradoxically required to do extra testing in some rather than miss even one case of a life-threatening illness.

In general, we accept this additional ‘cost’ of decision rules when their sensitivity is better than that of physicians, or, if just as good, the rule results in fewer unnecessary tests (higher specificity) than if the physician was working on their own. However, when that is not the case, we need to question their utility.

The results of the paper by Babl and colleagues raise such a question.1 Using a population of over 20 000 children prospectively enrolled in 10 EDs in Australia and New Zealand, researchers from the PREDICT (Paediatric Research in Emergency Departments International Collaborative) network validated a paediatric decision rule for head injury based on Nexus II criteria (removing only the age criteria) for the presence of clinically important intracranial injury. The incidence of significant head injuries was 19.2%. The rule performed very well: sensitivity 99% (95% CI 97.3% to 99.7%) and specificity (47.2% (95% CI 46.5% to 47.9%)).

However, with equivalent sensitivity (100% actually, 95% CI 99.0% to 100.0%), in actual practice, the physicians did far fewer scans than the rule suggested, yielding a much higher specificity (92.0%, 95% CI 91.6% to 92.3%) than the rule. It is worth noting this study was conducted as a secondary analysis of a prior validation of other rules, although all the data were originally captured prospectively. Nevertheless, the fact that the physicians were equally sensitive but more specific in detecting significant intracranial injuries should lead us to reconsider whether we should ask physicians to follow such a rule.

A second concern about decision rules is that they are often derived in highly selected populations and consequently exclude certain vulnerable patient groups. For example, the Ottawa knee and ankle rules exclude patients over 55 years; the Canadian and Nexus head injury rules for adults exclude patients who are anticoagulated. When we have a patient in front of us who was not included in the decision rule derivation, we are left to make our determination on the correct course of action. However, nature (and presumably medicine as well) abhors a vacuum, and physicians, or expert groups, may reflexively react with a wild streak of conservatism, often performing in a more risk-averse fashion than if no decision rules had existed in the first place. An example (and one we have fallen into) is the tendency to X-ray the ankle of every individual over 55 years who has no features suggesting fractures according to the Ottawa rule but is excluded due to their age.

The most striking (and expensive) example of this is the recommendation by a number of societies to perform CT scans in all patients with head injury who are anticoagulated. There are even recommendations to scan twice for the rare complication of late bleeding.2 ,3 But where is the data? In fact, as described in the expert practice review by Mason et al, the AHEAD study, the largest prospective observational study of patients with head injury anticoagulated with warfarin, found little support for either of these practices.4 In this study, patients with a GCS of 15 and no worrisome signs had a risk of bleeding equivalent to those of patients who were not anti-coagulated.

An oft forgotten adage is the concept of pretest probability. Failure to address this concept will lead to decision rule failure. In the case of chest pain decision rules, invariably the clinician should at least suspect cardiac chest pain before applying the rule. In the absence of any suspicion, applying the rule to all patients presenting with atraumatic chest pain will lead to needless downstream testing (and the odd irritatingly elevated troponin). Conversely, as eloquently demonstrated by the Pulmonary Embolism Rule Out Criteria(PERC) , the clinician must understand that the rule cannot be applied to those patients they truly think have a PE (pretest probability greater the 15% is suggested). ‘I am concerned this lady has a PE but she is PERC negative so I will discharge her anyway.’ The expert witness would have a field day but clinicians often fail to understand this.

Finally, it is worth reminding ourselves, as well as our trainees, that sometimes decision rules exclude patients not because they are higher risk but because they are lower. Inclusion criteria for both the Canadian, New Orleans and Nexus head injury rules included only patients with loss of consciousness and/or amnesia for the event; a slip and fall at work where one remembers all events, despite how hard the concrete, would not be covered by these rules. This means that, for what is likely the majority of minor head injury patients we see, there is no guidance available. Yes, you are on your own (with the exception of a good deal of literature that is consistent in telling us what high-risk symptoms are.)

In short, it is important for us to recognise what the value of decision rules is or should be, and to be cognizant of how they were derived and validated, particularly with regard to inclusion and exclusion criteria and pretest probability. The absence of evidence should not immediately trigger a non-evidence-based recommendation to fill the gap but rather suggests that physicians should practise based on their knowledge, experience and a bit of gestalt.

References

Footnotes

  • Contributors Both authors contributed equally.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient consent Not required.

  • Provenance and peer review Not commissioned; internally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles