The problem

Clinical prediction models (CPMs) have become an integral part of emergency medicine, such as the Wells score for deep vein thrombosis and APACHE II to predict mortality in intensive care.1 2

One problem with CPMs is that their predictive performance can deteriorate with time. For example, in 2013, Hickey et al 3 investigated the temporal change in prognostic performance of a CPM called EUROSCORE examining mortality after cardiothoracic surgery. They found that the model became increasingly mis-calibrated over time, in the sense that the observed mortality proportion diverged from the predicted mortality over time.

This phenomenon is known as calibration drift and is a particular issue for models implemented in clinical practice as a mis-calibrated model leads to inaccurate risk estimation and therefore potentially wrong decisions being made. This is particularly troublesome when considered in a clinical pathway with risk thresholds: the wrong course of action may be triggered because a patient who was predicted to be high risk was in fact low risk. For example, a patient with suspected acute myocardial infarction is predicted to be high risk and is admitted for additional investigations but in fact the model was mis-calibrated and they were actually low risk. That could lead to patients receiving unnecessary treatment or investigations that are of little benefit to them, exposing the patient to additional risk and the system wasting valuable resources. At a system level when CPMs are incorporated into national standards, it could lead to inaccurate information being provided to senior decision-makers.