Article Text
PDF
Abstract
Background Acute chest pain is a frequent reason to attend an emergency room, and various instruments for calculating the probability of an acute coronary syndrome exist.
Objective To assess the safety and efficiency of all available instruments investigated in sample validation studies.
Methods A systematic review was conducted. Studies were identified describing the development of instruments and all subsequent validations in electronic databases and reference lists of included studies. Inclusion was screened for, full papers checked and data extracted on salient clinical features, performance characteristics and quality in duplicate.
Results Of 20 derivation studies, 10 were at least validated once in 14 validations including 26 488 patients. One study by Selker and colleagues was validated in six new patient series and studies by Goldman et al and the Kennedy et al were both validated in three new patient series. All other studies were validated less than three times. In four out of six validations of the Selker et al study, the sensitivity of the prediction rule was 98% or higher. The corresponding values for specificity ranged from 4% to 34%. All remaining prediction rules showed sensitivity values below 95% in all validations.
Conclusions No instrument assisting clinicians in the diagnostic investigation of patients with suspected acute coronary syndrome consistently fulfils the safety requirements of clinicians.
- Acute coronary syndrome
- systematic review
- clinical prediction rule
- accuracy
- cardiac care
- diagnosis
Statistics from Altmetric.com
Introduction
The management of patients presenting with acute chest pain is daily routine at emergency departments worldwide. From the clinical point of view, timely identification and referral of patients with a case of acute coronary syndrome is of paramount importance for patient well-being and survival.1–3 However, ruling out acute coronary syndrome also represents a major challenge. A large proportion of patients with acute chest pain remain in the emergency room for hours and undergo repeated electrocardiogram recordings and laboratory tests before a rule-out diagnosis is established and patients can be sent home again.4 5 Unnecessary occupation of examination space in emergency wards may reduce service quality, is inefficient and also represents a relevant driver of healthcare costs.
As early as 1980, Pozen and colleagues recognised this problem and developed a predictive instrument for use on a handheld programmable calculator, which computed a patient's probability of having acute myocardial infarction based on information from patient's history and electrocardiogram (ECG) results.6 Over the last two decades, researchers have built on Pozen et al's work and developed various other diagnostic instruments combining components of the history, physical examination and ECG results to provide an estimate of the probability of acute coronary syndrome.
Despite this, the majority of doctors are reluctant to apply these rules in daily practice. Lack of confidence in the accuracy of such instruments and concerns regarding missing a patient with a potentially life-threatening illness might be reasons for not applying them. They argue that the performance of a prediction rule developed in one sample of patients (the derivation study) will be poorer when compared to its performance in another, although similar, sample of patients (the validation study).
The aim of this study was to systematically review the results of validation studies of all available diagnostic instruments for myocardial infarction or acute coronary syndrome in patients with acute chest pain to see which instrument identifies patients without acute coronary syndrome most accurately.
Methods
Literature search
In a first phase we searched for derivation studies in the following databases; Ovid MEDLINE (Ovid version, from inception to May 2009), and EMBASE and Scopus (from inception to May 2009). We used the following search terms: acute coronary, syndrome, myocardial infarction, chest pain, dyspnoea, probability, triage, decision support techniques, prediction and prediction rules. The search was conducted with no restrictions to language or year of publication (see appendix 1 for search strategy for Medline). We also manually searched the bibliography of all studies ordered in full text.
In a second phase, after identifying the original derivation studies of prediction rules, we used these references to search in ‘ISI Web of Knowledge’ (http://apps.isiknowledge.com/) for corresponding validation studies. This database provides detailed information on how often and by whom a published paper has been cited. In this set of publications we thoroughly searched for validation studies and again for derivation studies we might have missed in the first search in Medline and EMBASE. We assumed that all validations of an existing prediction rule would cite the derivation study and further derivation studies would be identified.
Selection criteria
Derivation studies
We included studies that reported prospectively and retrospectively collected data at the time of admission to the emergency department or by general practitioners outside the hospital and the diagnosis of acute coronary syndrome (unstable angina pectoris or myocardial infarction) was confirmed by appropriate reference standards. We included only studies concerning diagnosis after a first assessment in the emergency room and excluded those studies reporting on the prognosis of patients with suspected acute coronary syndrome. In addition, we specified that for the development of the clinical prediction rule the following candidate predictors must have been assessed: location and/or quality of chest pain, duration of chest pain episode and findings in the ECG.
Study selection
Derivation studies
Two reviewers (JR and DS) independently screened the titles and the abstracts of all retrieved references to identify derivation studies of prediction rules. Full text versions were ordered for all publications classified by any one of the reviewers as potentially relevant.
Validation studies
Two reviewers (DS and JS) independently screened the titles and abstracts of all references to identify validation studies. Full text versions were ordered for all potentially relevant publications.
Data extraction
We developed article review forms that were pilot tested and revised before use. Two reviewers (JR and JS) independently recorded details about inclusion criteria, age of patients, number of participating patients, study site, applied reference test to confirm or rule out an acute coronary syndrome, statistical methods applied, number of cases and results in a predefined form (available on request). We contacted authors of the selected studies for further information about missing data, but did not get any response.
Assessment of methodological quality
We assessed a study's quality with the validated Quality Assessment of Diagnostic Accuracy Studies (QUADAS) instrument.7 Two reviewers (DS and JS) applied the questionnaire on included studies independently. Answers were dichotomised as ‘yes’ or ‘no/unclear’. Disagreements in the assessment were resolved by consensus.
Data synthesis
Sensitivity and specificity (test accuracy) were calculated from two-by-two tables. We assessed heterogeneity of test parameters available from derivation studies using the ‘metan’ command implemented in the Stata V.11 software package (StataCorp, College Station, Texas, USA).
In publications where the two-by-two tables could not be reconstructed we report the results as area under the receiver operator curve. The probability threshold was set at 10% to calculate the test accuracy in the derivation study from Selker et al.8 The same value was used to calculate the accuracy of the validation studies by Selker et al, Seyal et al and Miller et al.8–10 In the study from Mitchell et al11 the probability threshold was set at 2% and Kellett12 reported the accuracy data for a threshold at the fourth decile.
Results
Derivation studies
Our searches in various databases identified 2710 records. After reading titles and abstracts we excluded 2573 papers. From the remaining 137 papers we excluded another 118 papers based on the full text assessment, because they did not report on diagnostic prediction rules as previously defined or contained no original data. A total of 20 papers were finally included in the list which we used to search for validation studies.6 8 13–30 (See figure 1)
Study flow diagram.
Validation studies
Our searches identified 1037 potentially relevant validation studies of the 19 derivation studies. After reading titles and abstracts we excluded 893 papers. From the remaining 144 papers we retained 14 after reading the full text.
Of 20 derived prediction rules, 10 had been validated at least once.6 8 17 21 22 24 26 28 31 We found 14 validation studies.8–12 17 21 22 26 28 31–34 The study by Selker and colleagues was validated in six new patient series and the Goldman et al and the Kennedy et al studies were both validated in three new patient series. All other studies were validated less than three times. One group17 reported on the validation of five derivation rules6 24–26 31 and Kennedy and Harrison22 validated the derived rule in three other hospitals. Details are shown in table 1.
Study characteristics
In the 10 included derivation studies 10 359 patients were included; the median number of included patients was 753 (87–3453, mean 1036, SD 950), the mean age of included patients ranged between 55 to 62 years, the number of variables included in the final model varied between 425 and 2522, prevalence of acute coronary myocardial infarction or acute coronary syndrome ranged from 11% up as high as 75%. With the exception of two studies17 28 all were conducted in USA. In two derivation studies recursive partitioning method was applied,26 31 the others used logistic regression analysis. Sensitivity and specificity across studies were heterogeneous (p<0.001).
The 14 validation studies included 26 488 patients, the median number of patients was 644 patients (mean 1614 SD 2822, range 108–11 991) and the mean age of participants was above 50 years in all studies. There were 3 temporal validation studies,8 28 34 and 11 geographical validation studies.9–12 17 21 22 26 31–33 A total of 10 studies were performed in the US, in 1 study patients from the US and Asia were included and 3 studies took place in European countries. In all studies the reference standard to rule in acute coronary syndrome was a combination of symptoms, changes in ECG and an increase in cardiac specific enzymes (creatine kinase (CK) in two studies, CK and CK-MB in nine and CK, CK-MB and troponin in three). The median prevalence of acute coronary syndrome or myocardial infarction was 42% (range between 2% and 92%). A total of 13 studies took place in emergency departments and 1 in a prehospital primary care setting. The outcome was myocardial infarction in seven studies and acute coronary syndrome (myocardial infarction, unstable angina pectoris) in seven studies. Details are shown in table 1.
Assessment of methodological quality
In general the quality of the studies was moderate. Related to the time the studies were performed, appropriate reference tests were applied in all studies. In only three studies was troponin part of the reference test.10 11 22 In three studies8 26 33 the reference test was described as being interpreted without the knowledge of the result of the prediction rule. The QUADAS score, applied to quantify the methodological quality of diagnostic tests, ranged between 8 and 11. The highest achievable value in this score is 14 points. In one derivation study,26 in 99.2% of all patients the diagnosis was verified by the reference test (96.3% in the corresponding validation study). In the other studies authors reported that the reference test was performed in all included patients. Furthermore, we checked, as recommended by Stiell and Wells,35 for the assessment of intraobserver or interobserver reliability of variables. Results about such assessments were not reported in any of the studies. Details are shown in table 1.
Performance of prediction rules
The performance of the prediction rules was reported in terms of sensitivities and specificities, except in one case.17 The sensitivities of the prediction rules in the derivation studies varied between 81% and 100%, and the specificities between 17% and 93%. In general, all validation studies showed lower values for the sensitivities compared to those in the corresponding derivation studies. The negative likelihood ratios ranged between <0.01 and 0.7.
In the three validations8–10of the Selker et al8 study applying the same probability threshold of 10% to calculate test accuracy data, the sensitivity of the prediction rule was between 95% and 98%; hence two fulfilled the false negative rate threshold of 2%, as set by doctors. In one study11 with a probability threshold at 2%, sensitivity was 100% (95% CI 92% to 100%), and in another12 (threshold fourth decile) 84% (95% CI 78% to 88%). The corresponding values for the specificity ranged from 6% (sensitivity 100%) to 84% (sensitivity 84%). The corresponding likelihood ratios are shown in table 2. Only one prediction rule22 has been developed since troponin is part of the reference standard test, and only four validations were performed10–12 22 applying the current reference standard.
Sensitivity and specificity of derivation and validation studies
All remaining prediction rules showed values of sensitivity above 95% in the derivation phase but sensitivities decreased to values below 95% in all validation studies, implying that more than 5% of patients with an acute coronary syndrome would be missed when doctors apply these diagnostic tools. Details about the results are given in table 2.
Variables in the final model
Information about the ECG was included in all final forms of the prediction rules, whereas location and/or quality of pain was part of eight6 8 17 22 24 26 28 31 of the rules and duration of the pain episode was one of the variables in only four22 26 28 31 of the prediction rules.
Discussion
Of the 10 developed prediction rules for the calculation of the probability of acute coronary syndrome (ACS) after initial clinical examination, ECG and enzyme measurement, 4 achieved a false negative rate of less than 5% in the derivation studies but performed less accurately when validated in different patient populations. The rule developed by Selker and colleagues8 was the ‘most safe’ with a false negative rate of 2% or less in most validations, but was not very efficient. Out of 100 patients with acute chest pain but without acute coronary syndrome, an acute coronary syndrome would be ruled out in only 4–34 patients when consistently applying the rule. This small yield limits the clinical impact and the potential for cost savings of this instrument.
Our results are in accordance with a review published recently by Hess et al36 who excluded prediction rules requiring computers to calculate the probabilities. Only eight studies, five derivation and three validation studies, satisfied their inclusion criteria. They concluded that all the assessed prediction rules had substantial methodological limitations and cannot be implemented into clinical practice.
Although we performed a thorough search in different databases we may have missed published derivation studies. Clinical prediction rules are variably indexed in the literature as clinical prediction rules, algorithms and risk scores, and thereby difficult to find in electronic databases. To overcome this limitation we carefully checked the reference lists of the included papers, review papers and medical textbooks. We are confident that we identified all relevant validation studies by screening the citations in ISI Web of Science for all included derivation studies. A strength of our review is that we included all published prediction rules, including those that need a computer for calculating the probabilities. Calculators/computers are nowadays available in most emergency rooms.
The majority of patients attending an emergency room because of chest pain have no acute coronary syndrome. A total of 30 years of research, including thousands of patients, has contributed to solving this clinical problem only to a minor degree. The challenge is to identify patients without acute coronary syndrome after a first examination including ECG and measurement of cardiac enzymes. In this respect, the early dreams of Pozen et al about efficient triage have not materialised so far. The reasons behind this might be complex.
From a methodological point of view various concerns arise. Some of the indicators such as eg, chest pain are very difficult to operationalise adequately. Arguably clinicians are not confident in relying on such information and will probably misclassify as present as a measure of precaution. This would explain the low specificity of many instruments. Variability in expressing chest pain could be another major source of inconsistency between studies. Moreover, the form of the regression models includes some tautology. Some of the definitional elements of myocardial infarction and acute coronary syndrome are concurrently assessed as candidate indicators for the instrument, which could inflate sensitivity.
A second concern refers to the reference standards used in the derivation and validation studies. A single study22 was derived and only a few studies10 11 22 were validated in the troponin era. This might confine the validity of earlier performed studies and limit their application in daily practice.
A third reason may pertain to the way doctors think. Doctors are predominantly socialised in an idealised zero-error environment and accept only very small or even no uncertainty.37 In a survey among emergency doctors most declared that a prediction rule would be helpful to calculate probabilities of an acute coronary syndrome. However, they would use it only when the false negative rate would be below 2%. Fear of a bad professional reputation or of legal problems might be an explanation.38
The 2% threshold represents the result of one survey questioning doctors explicitly about an acceptable false negative rate. The given answer is coherent with desirable zero-error medicine, but it is questionable if the 2% really reflects the level of uncertainty doctors intuitively accept in real practice. Notably, in instances in which another explanation of acute chest pain (that is, musculoskeletal pain) appears to be much more plausible, they rule out an acute coronary syndrome, despite a still persisting small probability of this being the correct diagnosis.
The successful introduction of prediction rules into clinical practice depends on the accuracy of the instrument and the settlement of sensible thresholds to either rule out or rule in a particular illness. Further studies should focus on the safety, efficiency and the actual impact on the management of updated or new developed prediction rules to rule out an acute coronary event. The introduction of a prediction rule in the diagnostic investigation of patients with suspected pulmonary embolism, also a potentially life-threatening illness, could be taken as a model. In a recent publication Roy et al39 demonstrated that the use of a prediction rule improved diagnostic decisions. They set a probability of less than 5% as appropriate to rule out pulmonary embolism.
In conclusion, we identified no instrument assisting clinicians in the diagnostic investigation of patients with suspected acute coronary syndrome that consistently fulfilled the safety requirements of clinicians in almost all validation studies. The prediction rule showing the most promising results is the one developed by Selker et al.8 However, its clinical impact is questionable because as a consequence of low values for specificity it has only limited potential of correctly ruling out acute coronary syndrome after a first examination of patients with acute chest pain.
Appendix 1 Search strategy for Medline (PubMed)
| Search no. | Queries | No. of hits |
| 1 | ‘Decision Support Techniques’ (MeSH) | 51438 |
| 2 | ‘Probability’ (MeSH) | 631435 |
| 3 | ‘Triage’ (MeSH) | 5033 |
| 4 | ‘Dyspnea’ (MeSH) | 10583 |
| 5 | ‘Chest Pain’ (MeSH) | 39672 |
| 6 | 4 OR 5 | 49572 |
| 7 | ‘Myocardial Infarction’ (MeSH) | 116433 |
| 8 | ‘Acute Coronary Syndrome’ (MeSH) | 382 |
| 9 | 7 OR 8 | 116747 |
| 10 | 1 OR 2 OR 3 | 676904 |
| 11 | 10 AND 9 AND 6 | 2710 |
References
Footnotes
Linked articles 098913.
Funding Helmut Horten Foundation.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.