Objectives To directly compare the diagnostic accuracy of four decision aids (Troponin-only Manchester Acute Coronary Syndromes (T-MACS), History, ECG, Age, Risk factors and Troponin (HEART), Thrombolysis in Myocardial Infarction (TIMI) and Emergency Department Assessment of Chest Pain (EDACS)) used to expedite the early diagnosis of acute coronary syndromes (ACS) in the ED.
Methods We prospectively included patients who presented to 14 EDs in England (February 2015 to June 2017) with suspected ACS within 12 hours of symptom onset. Data to enable evaluation of the T-MACS, HEART, TIMI and EDACS decision aids (without recalibration) were prospectively collected, blinded to patient outcome. We tested admission blood samples for high-sensitivity cardiac troponin I (hs-cTnI; Siemens ADVIA Centaur). Patients also underwent serial cardiac troponin testing over 3–12 hours. The target condition was an adjudicated diagnosis of acute myocardial infarction (AMI). We also evaluated the incidence of major adverse cardiac events (including death, AMI or coronary revascularisation) at 30 days. Diagnostic accuracy of each decision aid and hs-cTnI alone (using the limit of quantification cut-off, 3 ng/L) was evaluated by calculating sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV).
Results Of 999 included patients, 132 (13.2%) had AMI. C-statistics were 0.96 for T-MACS, 0.78 for HEART and 0.69 for TIMI. The sensitivities of T-MACS, HEART, TIMI, EDACS and hs-cTnI <3 ng/L for AMI were 99.2% (95% CI 95.7% to 100.0%), 91.8% (85.0% to 96.2%), 97.5% (92.9% to 99.5%), 96.2% (92.2% to 99.4%) and 99.2% (95.9% to 100.0%), respectively. The respective strategies would have ruled out 46.5%, 34.9%, 19.4%, 48.3% and 28.8% patients. PPVs for the decision aids that identify ‘high-risk’ patients were 80.4% (T-MACS), 51.9% (TIMI) and 37.2% (HEART).
Conclusions In this study, T-MACS could rule out AMI in 46.5% patients with 99.2% sensitivity. EDACS could rule out AMI in 48.3% patients with lower sensitivity, although the difference was not statistically significant. The HEART and TIMI scores had lower diagnostic accuracy.
- acute coronary syndrome
- cardiac care, diagnosis
Statistics from Altmetric.com
What is already known on this subject
The diagnosis of acute myocardial infarction (AMI) could now be ruled out after a single blood test in patients who have undetectable concentrations of cardiac troponin using a high-sensitivity assay.
Several decision aids including the HEART score, T-MACS, TIMI score and EDACS score have also been validated to rapidly rule out AMI in the Emergency Department.
What this study adds
We present what is, to our knowledge, the first direct comparison of the T-MACS, HEART, TIMI, EDACS scores and a troponin-only strategy to rule out AMI with a single blood test in the ED
T-MACS had the overall diagnostic accuracy for AMI (sensitivity 99.2%) and could have allowed early discharge for 46.5% of patients without serial blood sampling.
EDACS was the most efficient score (allowing early discharge for 48.3% patients) but had slightly lower (96.2%) sensitivity for AMI.
High-sensitivity cardiac troponin (hs-cTn) assays have greatly enhanced our ability to rapidly rule in and rule out a diagnosis of acute myocardial infarction (AMI) for patients attending the ED. Because of the time that was required for cardiac troponin concentrations to rise above the upper reference limit of contemporary assays, patients would previously undergo serial cardiac troponin testing over 6–12 hours as a routine. This led to unnecessary hospital admissions, which were expensive and inefficient for health services, contributed to the growing problem of crowding and led to substantial anxiety and inconvenience for patients and their families.
Taking advantage of the superior analytical characteristics of hs-cTn assays, several algorithms have been developed to accelerate the diagnostic pathway. Most of these still involve serial hs-cTn testing but over a shorter period than was previously necessary. These include, for example, the 0/1 hour algorithm, which can rule in and rule out AMI based on a combination of the initial hs-cTn concentration and the absolute change in hs-cTn concentration observed after 1 hour.1 2 However, it is also now possible to rule out the diagnosis of AMI in a minority of patients following a single blood test at the time of presentation to the ED. This relies on ‘ruling out’ AMI in patients who have hs-cTn concentrations below a very low threshold set at the limit of detection (LoD) of the assay3 4 or marginally above.5
All of these algorithms rely purely on hs-cTn concentrations and take no account of a patient’s symptoms, physical examination findings or prior medical history. There may be advantages of incorporating these data in a rapid diagnostic algorithm. Indeed, several decision aids have been validated for this purpose. These include (1) data-driven decision aids derived by multivariate analysis, such as the Troponin-only Manchester Acute Coronary Syndromes (T-MACS) decision aid and Emergency Department Assessment of Chest Pain (EDACS) score; (2) the History, ECG, Age, Risk factors and Troponin (HEART) score, which was developed based on expert knowledge; and (3) scores developed to risk stratify patients with confirmed acute coronary syndromes (ACS) such as the Thrombolysis in Myocardial Infarction (TIMI) risk score, which have then been applied to the undifferentiated population in the ED.
The diagnostic accuracy of these four decision aids has never been directly compared. We therefore aimed to determine the diagnostic accuracy of T-MACS, EDACS, HEART and TIMI in patients presenting to the ED with suspected ACS, using a single blood test drawn at the time of arrival in the ED.
Design and setting
This study was nested within the Bedside Evaluation of Sensitive Troponin (BEST) study, a prospective diagnostic accuracy study that recruited participants at 18 centres in England. The BEST study had six work streams. The work stream presented here was labelled the Sensitive Troponin for Admission Reduction study, and for this analysis we used data from an evaluation of the Siemens ADVIA Centaur High-Sensitivity Cardiac Troponin I assay, which included data from 14 of the 18 participating centres (listed in the online supplementary appendix).
We included patients who presented to the ED with pain or discomfort in the chest, epigastrium, arms, shoulders or neck that did not have an apparent non-cardiac cause and in whom the treating clinician suspected a diagnosis of ACS. Patients were excluded if they were unable or unwilling to provide written informed consent, if they had obvious evidence of ST elevation myocardial infarction, if they had another medical condition that would require hospital admission (thus excluding patients with type 2 AMI) and patients with peak symptoms >12 hours before presentation.
Data collection and laboratory analysis
The data for calculation of the T-MACS, HEART, TIMI and EDACS decision aids were prospectively recorded by the treating clinicians using a bespoke case report form. To minimise missing data, research nurses were permitted to record data on a patient’s demographics, prior medical history, family history, smoking history and physiological parameters. However, all sites were instructed that only the doctor responsible for the patient’s care could record data that required clinical judgement (including ECG interpretation, suspicion of ACS and the presence or absence of worsening angina).
We took blood samples from participants at the time of arrival in the ED. Within 4 hours, serum was extracted and frozen. The serum was frozen at −40°C or below for up to 4 weeks and at −80°C thereafter, until analysis. Previously unthawed samples were then analysed for high-sensitivity cardiac troponin I (hs-cTnI) (Siemens ADVIA Centaur hs-cTnI) in batches. The assay has an overall 99th percentile upper reference limit of 47 ng/L (57 ng/L in men and 37 ng/L in women); the coefficient of variation is <10% at 6 ng/L; the LoD is 1.6 ng/L; and the limit of quantification (LoQ), which is defined as the lowest concentration with coefficient of variation <20%, is 2.5 ng/L.6 The manufacturer recommends that hs-cTnI concentrations are only reported down to the LoQ (2.5 ng/L).7 Based on the International Federation of Clinical Chemistry Committee for Cardiac Biomarkers recommendation to report hs-cTn assay concentrations in ng/L and in integers, the lowest reportable concentration with this assay is therefore 3 ng/L.8 This hs-cTn assay is CE-marked and approved by the US Food and Drug Administration and at the time of writing is being assessed by the National Institute for Health and Care Excellence in England.
A summary of the variables included in each of the decision aids is shown in figure 1. Each of the four decision aids was applied in the manner that has previously been reported in the literature, without recalibration. We calculated the T-MACS decision aid as has been previously described, using a refined formula that is used in current clinical practice.9 This calculates the probability (p) of ACS as follows: p=1/(1+e-(-4.65 + 1.828a + 1.54b + 0.849c + 1.783d + 1.878e + 1.412f + 0.084g)). Where a denotes acute ECG ischaemia; b denotes a pattern of worsening (or crescendo) angina; c is pain radiation to the right arm or right shoulder; d is pain associated with vomiting; e is visible diaphoresis in the ED; f is hypotension (defined as systolic BP <100 mm Hg); and g is cTn concentration. For all variables except g, a value of ‘1’ is entered if the feature is present and a value of ‘0’ is entered if it is absent. Patients are then assigned to risk groups as follows: p<0.02, ‘very low risk’ (rule out ACS); 0.02≤p<0.05, ‘low risk’ (suitable for serial hs-cTn testing in a low dependency environment such as an ED observation ward or ambulatory care unit); 0.05≤p<0.95, ‘moderate risk’ (suitable for serial hs-cTn testing in a general medical ward); p≥0.95, ‘high risk’ (rule in ACS).
The HEART score was also calculated as described in previous research,10 although data on ‘obesity’ were not collected and the troponin criteria were modified. To avoid using unrealistically high cut-offs, an hs-cTnI below the LoQ scored 0 points; between the LoQ and the sex-specific 99th percentile scored 1 point and above the sex-specific 99th percentile scored 2 points. The ‘History’ element of the HEART score, which is subjective, was recorded by asking the treating clinician to respond to the following question: ‘In my opinion, for an acute coronary syndrome these symptoms are: Highly suspicious; moderately suspicious; or slightly suspicious’. The TIMI score was calculated in accordance with previous work in this setting.11 ‘Known coronary stenosis’ was replaced by a prior medical history of coronary artery disease, which included myocardial infarction or previous coronary intervention (percutaneous or coronary artery bypass grafting). Finally, the EDACS score was also calculated in line with the original derivation study.12 ‘Diaphoresis’ was considered to be diaphoresis reported by the patient, in accordance with that report. For all analyses, we used the sex-specific 99th percentile of the hs-cTnI assay. The EDACS score was originally designed to be used with serial cardiac troponin testing over 2 hours. However, as this study aimed to compare single test strategies, we only considered the hs-cTnI concentration measured at the time of arrival in the ED.
Target condition and reference standard
The target condition for evaluation of diagnostic accuracy was an adjudicated diagnosis of type 1 AMI. We defined AMI in accordance with the third universal definition, which was current at the time of adjudication. By this definition, patients met the criteria for AMI if they had a rise and/or fall of cardiac troponin with at least one concentration above the 99th percentile upper reference limit of the assay. In addition, patients were required to have either symptoms compatible with myocardial ischaemia, relevant ECG changes, imaging evidence of new loss of viable myocardium or angiographic evidence of coronary thrombus.13
The reference standard for AMI diagnosis was cardiac troponin concentrations, measured on arrival and at least 3–6 hours after arrival. The study protocol required that all patients should undergo serial cardiac troponin testing over 3 hours if a hs-cTn assay was in use, or 6 hours if a contemporary assay was used in practice at the relevant centre. The cardiac troponin assay that was used in clinical practice at the time of the study was used for AMI adjudication (see online supplementary file 1). No site used the Siemens ADVIA Centaur hs-cTnI assay during the study period. Adjudication of AMI was undertaken by two investigators acting independently, blinded to results of the investigational hs-cTnI assay and to the T-MACS, HEART, EDACS and TIMI decision aids.
We also followed patients up after 30 days by telephone and medical record review. If a patient was persistently uncontactable, we contacted their general practitioner for information. In the UK, details of all interactions with secondary healthcare providers are communicated to the patient’s general practitioner. These follow-up data were used to evaluate the prognostic outcome of major adverse cardiac events (MACE). MACE included AMI (both prevalent and incident), death (all cause) and coronary revascularisation.
A statistical analysis plan was created prior to commencing analysis, in accordance with Standards for Reporting Diagnostic Accuracy Studies guidance.14 We summarised baseline characteristics, diagnoses and prognostic outcomes using descriptive statistics. Diagnostic accuracy of the four decision aids being evaluated and of hs-cTnI alone using the LoQ cut-off (3 ng/L) was evaluated by constructing 2×2 tables, which were used to calculate test characteristics including sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) with respective 95% CI. Overall diagnostic accuracy was evaluated by receiver operating characteristic (ROC) curve analysis using the non-parametric method described by DeLong et al.15 We compared sensitivities and specificities by McNemar’s test for paired data. For these pairwise comparisons, we included patients with complete data for both decision aids being compared. Because of the nature of this analysis, we undertook a complete case analysis, excluding patients with missing data to permit calculation of the outcome of the relevant decision aids. We cross-tabulated data and constructed ROC curves using SPSS V.23.0. We compared the areas under ROC curves and calculated test characteristics using MedCalc V.18.104.22.168.
Sample size for the BEST study was driven by the required precision for estimates of diagnostic accuracy. We aimed to enrol 1575 patients to identify an algorithm with 90% specificity and 100% sensitivity, such that the lower bound of the 95% CI for sensitivity did not cross 95% (assuming a prevalence of 10% and 5% loss to follow-up). For this study, the sample size was dictated by serum sample availability for analysis with the Siemens ADVIA Centaur hs-cTnI assay.
In total, 999 patients were included in this study between February 2015 and June 2017. Of those patients, 132 (13.2%) had an adjudicated diagnosis of AMI and a further 21 patients developed one or more MACE within 30 days (figure 2). Table 1 demonstrates the baseline characteristics of the included participants.
The test characteristics of each decision aid evaluated are shown in table 2. T-MACS had the highest sensitivity and NPV of all decision aids evaluated (99.2% and 99.8%, respectively) and the HEART score had the lowest (91.8% and 97.0%, respectively). This difference in sensitivity was statistically significant (p=0.004). T-MACS had equal sensitivity to a troponin-only strategy using the LoQ (3 ng/L) of the hs-cTnI assay as a ‘rule out’ cut-off (the ‘LoQ strategy’), but T-MACS would have allowed more patients to be immediately discharged (46.5% vs 28.8%, p<0.0001 for comparison of specificity).
Compared with T-MACS, only EDACS would have allowed a higher proportion of patients to be immediately ruled out (48.3% vs 46.5%, p=0.75 for comparison of specificity). This came at a cost of lower sensitivity for AMI (96.1% vs 99.2%, p=0.70). EDACS missed three AMIs compared with one with T-MACS. The patient with AMI that would not have been detected by T-MACS had an initial hs-cTn T concentration of 30 ng/L but repeat tests 3 and 6 hours later revealed concentrations of 4 ng/L on each occasion. The patient received no treatment, was discharged on the same day and experienced no further events within the 30-day follow-up period. Because no alternative reason could be established for the initial hs-cTn concentration, the patient was assigned an adjudicated diagnosis of AMI. At 30-day follow-up, four patients who were identified as very low risk by T-MACS had undergone coronary revascularisation, compared with none with EDACS and two with the LoQ strategy.
Figure 3 summarises the ROC analysis for each decision aid. As the outcomes of EDACS and the LoQ strategy are dichotomous (low risk vs not low risk), they could not be included in the ROC analysis. T-MACS had a substantially higher area under the ROC curve (AUC) than both the HEART score and TIMI score. The AUC for T-MACS was 0.96 (95% CI 0.94 to 0.98) compared with 0.78 (95% CI 0.74 to 0.83) for HEART (p<0.0001 for comparison with T-MACS) and 0.69 (95% CI 0.64 to 0.74) for TIMI (p<0.0001 for comparison with T-MACS).
The proportion of patients with AMI in each risk group for the relevant decision aids is shown in table 3. Again, T-MACS provided the greatest overall risk stratification, with 80.4% patients in the high-risk group having AMI. Meanwhile, the prevalence of AMI in the high-risk groups identified by the HEART score and TMII score was 36.2% and 51.9%, respectively. EDACS does not identify a high-risk group but the prevalence of AMI was 24.3% in those who were not identified as being low risk.
Using the Siemens ADVIA Centaur hs-cTnI assay, we have validated the performance of four decision aids for use in patients who present to the ED and have a suspected diagnosis of ACS. All four decision aids effectively risk-stratified patients. For early rule-out, the HEART score and TIMI score had sensitivities and NPVs that may be unacceptable to clinicians in practice. T-MACS, EDACS and the LoQ strategy all had NPVs >99%. However, the LoQ strategy would have ’ruled out’ fewer patients than either EDACS and T-MACS. EDACS had a lower sensitivity for AMI than T-MACS but more patients underwent coronary revascularisation after 30 days using T-MACS. T-MACS had the greatest overall diagnostic accuracy on ROC analysis and offered superior risk stratification to the other decision aids, with the highest positive predictive value for AMI among high-risk patients.
To our knowledge, this is the first direct comparison of these decision aids and is the first validation of each decision aid using this troponin assay. Validation of decision aids with each troponin assay is important because assays can have substantially different diagnostic accuracy and may interact differently with other variables. In previous work, Carlton et al compared the diagnostic performance of four decision aids with two hs-cTn assays: the Roche Elecsys hs-cTnT assay and the Abbott ARCHITECT hs-cTnI assay.16 Our findings are broadly similar. In that work, the HEART score had a sensitivity of 93.7% with an NPV of 98.3% when used with hs-cTnT (Roche Elecsys), and a sensitivity of 97.0% with an NPV of 99.3% when used with hs-cTnI (Abbott ARCHITECT). Meanwhile, the TIMI score had 100.0% sensitivity with hs-cTnT and 95.5% sensitivity with hs-cTnI.
A retrospective comparison of HEART and TIMI in 8255 patients from a registry found that HEART had a similar NPV to TIMI (98.2% vs 97.8%) but HEART had a greater AUC, which again agrees with our findings.17 In another retrospective evaluation of 118 822 patients, a modified HEART score and EDACS could both achieve an NPV of 99.5% using cardiac troponin cut-offs below the 99th percentile, although EDACS identified more patients (60.6%) for early discharge.18 Finally, in a randomised controlled trial of 558 patients, both the TIMI score and EDACS allowed approximately one-third of patients to be discharged within 6 hours of arrival in the ED, with no low risk patients experiencing MACE within 30 days.19
To date, there have been no direct comparisons of the accuracy of T-MACS with other decision aids.20 This work therefore makes a valuable contribution to the field. T-MACS was derived and validated by logistic regression to combine information from a patients’ symptoms and medical history with information from the ECG and cardiac troponin concentration on arrival in the ED. T-MACS uses those data to calculate the probability that each patient has an ACS, and uses that probability to stratify patients into four risk groups, each of which has a suggested course of action. In previous work, T-MACS has been shown to identify up to 40% patients as very low risk, facilitating immediate discharge, with >99% NPV for AMI.21–23 In addition, it has previously been shown to identify a high-risk group with ~90% PPV. There is an increasing evidence base for T-MACS, with successful validation using high-sensitivity,21 22 contemporary23 and point of care cTn assays.9
This current work suggests that T-MACS may have advantages over other decision aids in terms of diagnostic accuracy. Certainly, this work has established that T-MACS is a very reasonable alternative to other decision aids when used with the Siemens ADVIA Centaur hs-cTnI assay, and it may have some important advantages. For example, the calculation of a probability of ACS may be used to inform shared decision-making with patients, which has previously been shown to facilitate more judicious use of healthcare resources with no apparent impact on patient safety.24 T-MACS could also be used to guide treatment, enabling the automated and objective weighing of risks and benefits of different treatments including antiplatelet agents.25 Finally, when used in practice, the algorithm used to calculate T-MACS could be continually refined through machine learning to avoid calibration drift, the phenomenon by which diagnostic accuracy of decision aids tends to deteriorate over time. Even before these more advanced features are developed, T-MACS could be implemented alongside serial cardiac troponin testing algorithms to optimise the efficiency of the diagnostic pathway. A potential pathway for implementation is depicted in figure 4, whereby patients who are not ‘ruled in’ or ruled out using T-MACS after a single troponin test would proceed to have a second test in 1 hour.
We could not include all patients who had been enrolled in the BEST study due to lack of sample availability for analysis with this assay. This is a common problem in studies of this nature, and there is no reason to believe that this has biassed our findings. However, including all consenting patients in this analysis would have been preferable. We also excluded patients who had missing data for calculation of the decision aids evaluated. Measures were put in place to mitigate this risk, by employing a full-time data manager, emphasising the importance of high-quality data to the sites and encouraging early validation of data quality after enrolment. However, despite these measures there were still some missing data. It is notable that more data were missing for the HEART score (n=874) and TIMI Score (n=844). This may be because those scores incorporate a larger number of individual variables. For example, in order to have complete data for risk factors, patients required data for each of hypertension, hyperlipidaemia, diabetes mellitus (types 1 and 2), smoking history and family history of ischaemic heart disease. Thus a total of 82 patients were excluded for missing data about one or more risk factors. This may be because certain risk factors may not be routinely included in a focused ED history (eg, family history of ischaemic heart disease). Further, some clinicians may have been reluctant to commit to providing an opinion on symptom typicality, as these data were missing in 51 cases. Again, there is no reason to believe that this has biassed our findings but in future research it will be important to carefully design studies to minimise this risk.
In this study, the T-MACS decision aid had excellent overall diagnostic accuracy, effectively risk stratified patients, ruled out AMI in 46.5% patients with high sensitivity (99.2%) and NPV (99.8%), and ruled in AMI with ~80% PPV. The EDACS score ruled out AMI for 48.3% patients with slightly lower sensitivity (96.2%) and marginally lower NPV (99.3%), although the difference was not statistically significant. The HEART and TIMI scores had lower diagnostic accuracy.
We acknowledge the support of the National Institute for Health Research Clinical Research Network and all the staff at the recruiting centres who worked so hard to deliver this study.
Presented at European Society for Emergency Medicine (EUSEM) Annual Congress, 2018 (Glasgow)
Contributors All authors meet authorship criteria as specified by the ICMJE.
Funding Siemens provided reagents without charge for the purpose of this study. The study was sponsored by Manchester University NHS Foundation Trust.
Competing interests RB: Speaker fees from Singulex, Roche, Abbott, Siemens, Beckman, Ortho, ET Healthcare, Alere; Research grants from Roche, Abbott Point of Care and Singulex; donation of reagents for research from Roche, Siemens, Singulex, Abbott, Alere, FABPulous BV, Beckman, Abbott Point of Care.
Patient and public involvement statement We thank the Manchester Heart Centre Service User Group for their input into the design of this study.
Patient consent for publication Not required.
Ethics approval The study was registered on the National Institute for Health Research Clinical Research Network portfolio (reference UK CRN 18000) and ethical approval was obtained from the National Research Ethics Service (reference 14/NW/1344).
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available upon reasonable request.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.