Using clinical risk models to predict outcomes: what are we predicting and why?

Clinical risk prediction uses patient and disease characteristics to estimate the probability of a current diagnosis (diagnostic), future outcome (prognostic) or treatment effect (therapeutic), and thus assist clinical decision making.1 This paper focuses on prognostic clinical risk prediction of a future outcome. Increasing availability of large data sets linking clinical features to outcome data is creating huge opportunities to develop clinical prediction models. Data analytical methods, such as machine learning, increase the sophistication of modelling techniques but may limit the clinician’s ability to interpret the risk that the model is predicting. If we want to use clinical risk prediction to improve care, we need to understand how prediction works and what it is predicting.

How can we use clinical risk prediction?

Clinical risk prediction typically uses a mathematical model to predict the risk of an adverse outcome based on clinical measurements or the presence of clinical features.1 Clinicians can then use the predicted risk to direct or assist decision making.2 The model can be simplified into a score, with a higher score predicting a higher risk of adverse outcome. Table 1 lists some commonly used scores in emergency medicine. Patients with higher risk are typically directed towards intervention (admission, investigation or specialist referral) or monitoring to determine the need for intervention, while those with lower risk may avoid intervention. This is presumably based on the assumption that intervention will reduce the risk of adverse outcome. However, as emergency care systems increasingly manage people with more severe frailty and multiple long-term conditions, we should question this assumption. If the risk of adverse outcome relates to underlying frailty or long-term conditions, then intervention directed at the acute pathology is unlikely to be effective. We therefore need clinical risk prediction to identify who will benefit from intervention, not just who is at risk of adverse outcome.

It may be argued that the role of emergency medicine is simply to identify risk of adverse outcome and leave others to determine whether or how the risk can be reduced. This rather impoverished view of the specialty neglects the harm associated with futile intervention, raising unrealistic expectations, loss of independence and social support following unnecessary admission, and the knock-on problems of access block and ED crowding.

How are clinical prediction models developed?

Understanding how clinical prediction models are developed can help us understand how they work. The process of developing a clinical prediction model is relatively simple and involves sampling the relevant patient group, recording potential predictor variables, and then recording outcomes over a specified follow-up period. Statistical methods are then used to identify the strongest independent predictors of outcome and combine them into a model.1

Measuring the association between predictors and outcome is simple in principle, but encounters a substantial practical problem, in that patients in the study receive treatment to reduce their risk of adverse outcome. If treatment is effective, then those who benefited from treatment will avoid adverse outcome and will not be predicted by the model. This means that predictive performance for avoidable adverse outcome may be underestimated, important predictors for guiding treatment may be missed and the model may fail to predict cases that will benefit from treatment. Conversely, if adverse outcomes occurred despite treatment, the model may be developed to predict unavoidable adverse outcomes where treatment is futile.

Box 1 describes examples where treatment effects may undermine the value of clinical prediction models for identifying patients who will benefit from intervention. Evidence that treatment effects undermine clinical prediction models may be limited but logic suggests that using clinical prediction scores developed on a treated population to guide treatment decisions must be problematic. Either treatment works, in which case treatment effects cause the problems outlined above, or it does not, in which case there is little rationale for treating those at highest risk.

This issue is specific to prognostic risk prediction of future outcomes (the focus of this paper). In diagnostic risk prediction, the reference standard diagnosis is already present. However, developers and users of clinical risk prediction must be clear whether prediction is diagnostic or prognostic. For example, we could use the Emergency Department Assessment of Chest Pain Score (EDACS, see table 1) to diagnose an acute coronary syndrome or predict future adverse cardiac events, but using it to do both risks confusion.

How can we address this problem?

If we understand a clinical prediction model and the research used to develop it, then we can use the model better to improve patient care. The following examples show how we can examine the model or score to gain insights about how it works in practice.

How can we make better use of clinical prediction scores?

Clinical prediction scores can support but should not dictate decision making. This means that guidelines should not use clinical prediction scores to direct patient management but may recommend calculating a score to inform clinical judgement. If we use a score to estimate the risk of an adverse outcome, then we should recognise that this probably represents the risk for a treated patient, so we should not give false reassurance that intervention will eliminate the risk. If the tool suggests a significant risk of adverse outcome, then we should use clinical judgement to determine whether the risk is related to acute illness or underlying frailty and long-term conditions. We should then be honest and realistic about the potential for intervention to reduce the risk.

Making better use of clinical prediction tools requires better evidence. Studies of clinical prediction models should report prediction for interventions and outcomes separately, so readers can judge how well the model predicts intervention and how each predictor is associated with intervention and outcome. Studies could use expert adjudication to determine whether outcomes were avoidable or were avoided through intervention. Finally, big data and data analytics can produce models that predict adverse outcomes with impressive accuracy and narrow CIs, but we still need to understand how prediction works and whether treatment can prevent adverse outcomes. This may be difficult if data analytics researchers do not describe their complex models in a transparent manner. Box 2 summarises the key messages for users and developers of clinical prediction scores.