The diagnosis of acute coronary syndromes (ACS) rests on clinical history, changes on the ECG, and cardiac marker protein data. Each of these evolves after presentation and is modified by treatment. Marker protein measurements provide definitive diagnostic and prognostic information but take several hours after the onset of symptoms to become positive. This has led to the development of protocols in chest pain units in many centres to manage patients in the early hours after the onset of symptoms and before a definitive diagnosis can be made.^1–6 A large proportion of patients who present to emergency departments with chest pain have non-cardiac diagnoses and most of these patients are most appropriately discharged directly home. In practice, a small but significant proportion of patients are sent home inappropriately,⁷ leading to potentially serious clinical errors and litigation. On the other hand, many relatively low risk patients are inappropriately admitted to telemetry and high dependency units to rule out acute cardiac ischaemia.⁸ In the centres used for this study, around 2% of patients were inappropriately discharged from emergency departments, whereas about 30% of patients presenting with acute chest pain were admitted with possible ACS but ultimately had the diagnosis ruled out.

Better use of clinical and ECG information available at presentation can improve identification of patients with evolving ACS. This has the potential to improve clinical care, since many triage and treatment decisions have to be made early, and to optimise use of resources, including chest pain units. Studies confirming that clinical, as well as ECG, factors are highly discriminatory for evolving ACS have strengthened research in this area recently.^9–11 Various statistical and computer based methods have been used to analyse clinical and ECG data from patients with chest pain with a view to improving identification of high risk patients at presentation. These methods include logistic regression,^8,12–15 classification trees,^16,17 and artificial neural networks (ANNs).^18–20 Each of these methods has advantages and disadvantages, although, suitably optimised, they can all provide accurate classification of low and high risk patients from data available at presentation.²¹ We used logistic regression in this study. This is a non-linear classification technique that uses binary and continuous data to derive a series of coefficients, which, when applied to previously unseen samples, yield a probability of a single output (for example, the presence of ACS).

Our previous study suggested that a simple logistic regression model based only on ECG data performed almost as well as a more extensive model incorporating clinical data items.¹² The goal of that study was to develop a predictive model for myocardial infarction (MI), whereas the present study aimed at identifying the broader range of coronary syndromes. Selker et al⁸ have described a simple logistic regression model based mainly on ECG data, the acute cardiac ischaemia-time insensitive predictive instrument (ACI-TIPI), to identify patients with acute cardiac ischaemia. Use of ACI-TIPI in 10 US hospitals increased the rate of discharge while decreasing inappropriate admission to high dependency beds. Other studies have also shown the potential for decision aids to improve admission and discharge practices for patients with acute chest pain.^16,22–24 To gain widespread acceptance, a model should be easy to use in the emergency room, discriminate between low and high risk patients with a high degree of accuracy, be well calibrated, perform robustly with data from different institutions, and operate in a way that is clinically meaningful. To date, no algorithm has been described that fully satisfies these criteria.

The goals of this study were, firstly, to derive and optimise logistic regression models to identify patients who are developing ACS by using clinical and ECG data from the time of presentation; secondly, to test these models on data prospectively collected from centres other than the one from which the training data were collected; and thirdly, to document the properties of the models in terms of performance, calibration, and robustness.

PATIENTS AND METHODS

Study design

This was a prospective cohort study of unselected patients with chest pain presenting to emergency departments of four UK teaching hospitals: the Royal Infirmary of Edinburgh (hospital 1); Western General Hospital, Edinburgh (hospital 2); Northern General Hospital, Sheffield (hospital 3); and Leicester Royal Infirmary (hospital 4). Data from consecutive patients presenting to hospital 1 were used to derive logistic regression models that were subsequently tested on data from the other three centres. Methods for data collection are identical to those previously described.¹² Ethical approval was obtained for the study at each site and informed consent was obtained form each patient participating in the study.

Study population

Clinical and ECG data were collected at presentation in the emergency departments of participating hospitals. Consecutive patients presenting with acute chest pain were recruited. Hospitals 1 and 2 are in the same city and serve a population of just over 500 000. The accident and emergency department of hospital 1 receives around 90 000 patients a year. During the four month period (August to December 1995) of data collection from this hospital, 4.2% of presentations were with acute non-traumatic chest pain. Hospital 2, serving the same population as hospital 1, receives medical emergencies through an acute assessment unit. It receives 25 000 patients a year, and during the period of data collection (February to August 1996), 10.1% of patients presented with chest pain. This high rate reflects the presence of a regional cardiac unit in hospital 2, and the high proportion of patients with diagnosed ACS reflects the fact that many patients with chest pain with less acute presentations in the city are seen in chest pain clinics and in a general practice assessment unit. Hospital 3 serves a population of 530 000 and has 75 000 emergency department attendances a year, 4% of which are for acute chest pain. Chest pain data from this hospital were collected over three months (September to December 1992). Data for a small sample of patients were collected from hospital 4.

Training data for the logistic regression models were obtained from 1253 consecutive patients aged 18 or over presenting with non-traumatic chest pain to hospital 1. The study included both patients who were admitted and those who were discharged. The attending doctors in the emergency department recorded clinical and ECG data on a purpose designed form. Three researchers—a consultant physician, cardiology registrar, and research nurse—assigned the final diagnosis for all patients independently. This diagnosis made use of follow up ECGs, cardiac markers, other investigations, and a clinical history obtained from the patient’s follow up notes. For patients discharged directly from the emergency department or for those with incomplete follow up, the patients or their general practitioners were contacted for information about diagnosis or continuing symptoms one month after initial attendance. Further data to test the models were obtained from the emergency medical units at hospital 2 (n  =  1268), hospital 3 (n  =  626), and hospital 4 (n  =  152). The methods for data collection and diagnosis were as described above. In each hospital, patients were recruited 24 hours a day, seven days a week.

Measurements

All patients admitted to hospital had serial cardiac marker measurements in line with local protocols. The rate of missed diagnosis of ACS among those discharged was very low (< 2%). Creatine kinase (CK) > 180 U/l for women and > 200 U/l for men was regarded as abnormal, as was CK-MB activity > 5% of total CK activity or a CK-MB mass > 8 μg/l. CK-MB mass was measured by a standard method (Behring Diagnostics). Troponin T or I was measured for patients admitted or regarded as being at high risk of ACS, and a concentration > 0.1 μg/l was regarded as abnormal.²⁵ ACS was diagnosed in all patients who had positive cardiac markers. MI was diagnosed on the basis of clinical history, serial ECGs, and cardiac markers in line with current recommendations.²⁶ ST segment elevation MI (STEMI) was diagnosed when ST segment elevation > 1 mm or pathological Q waves developed in two or more regional ECG leads. Non-ST segment elevation MI (non-STEMI) was diagnosed when positive cardiac markers were accompanied by changes (ST depression, T wave inversion) on sequential ECGs. ACS without MI was diagnosed when ECG changes not diagnostic of STEMI occurred in the absence of increased markers, when increased cardiac markers were not accompanied by ECG changes, when the patient had an unstable course necessitating acute cardiological intervention, when ST elevation of 1.5 mm of more was present on stress testing, or when the patient had an adverse cardiac event (death, MI, or need for urgent intervention) within 30 days of the initial event. Overall, 15% of patients in the study underwent stress testing.

Statistical models

For logistic regression, binary code was used to indicate presence or absence of factors and continuous valued variables were assigned to categorical ranges represented by binary indicator variables. Age was divided into three categories: ⩽ 40 years, 41–60 years, and > 60 years. Duration of symptoms was divided into four categories: ⩽ 4 hours, 5–12 hours, 13–48 hours, and > 48 hours. The second categories of age and duration were designated as reference categories.

The logarithm of the likelihood ratio (log₁₀ LR) for each variable was estimated from the training sample and ranked from high to low. Log₁₀ LRs for all other samples were computed and ranked according to the training sample to show any differences between groups. The formula for computing the log₁₀ LR of the jth binary variable, x_j, is given thus:

Note that some variables may exhibit a value of zero in either the numerator or denominator for a particular sample. In these cases, the log₁₀ LR was set to − 2.5 or + 25, respectively, to make the graphs meaningful.

Logistic models of the following form were computed:

where the summation is over the variables included in the model for some new individual, x. The maximum likelihood estimate of the coefficients, {b,w_i|i = 1,2,…}, was computed by the method of iteratively reweighted least squares.²⁷

To overcome sampling effects all experiments underwent 10-fold cross validation (resampling without replacement) and the results reported are based on the average (mean value) of the 10 model predictions after adjustment to correct differences in ACS prevalence between training and test samples. Such averaging is guaranteed to be no worse, on average, than the results from any single model²⁸ and can reduce variability in results without increasing bias. Receiver operating characteristic (ROC) curves were computed from 100 equally spaced threshold values. Their areas were computed by the trapezoidal method of integration and their standard errors were computed as suggested by Hanley and McNeill.²⁹ Computation were done within the Matlab environment (The MathWorks, Inc, Natick, Massachusetts, USA) version 6.5.0 with the freely available toolbox Netlab (www.ncrg.aston.ac.uk/netlab).³⁰

Sensitivity was defined as true positives/(true positives +false negatives), specificity as true negatives/(true negatives + false positives), accuracy as (true positives + true negatives)/total number of patients, positive predictive value as true positives/(true positives + false positives), and negative predictive value as true negatives/(true negatives + false negatives). Calibration of a model was the match between predicted and observed proportions of patients with ACS over the entire predictive range of the model. In combining model output with clinical opinion, ACS was diagnosed if the model output was positive or if the output was negative but clinical opinion favoured a major cardiac event. When troponin was positive, ACS was diagnosed whether the model was positive or negative. Diagnosis based on statistical models does not incorporate marker protein data but attempts to predict the final diagnosis for which marker protein data are available.

RESULTS

Training data and model derivation

The training (hospital 1) data were from 1253 patients with a mean age of 57.6 years (range 18–92 years); 829 (66.2%) were men. Table 1 compares univariate statistics for ACS versus non-ACS patients in this cohort. The final diagnosis was MI for 274 (21.9%), ACS for 466 (37.2%), and non-cardiac for 529 (42.2%) patients. Table 2 shows the 45 potential factors, along with their log₁₀ LR, for diagnosis of ACS. Figure 1A displays the inputs in identical order for comparison with hospital 2 (fig 1B) and hospital 3 test data (fig 1C). Clinical factors are as discriminatory as ECG items for diagnosis of ACS. Clinical data items with both positive and negative association with ACS are identified.

Variables were selected for logistic regression models on the basis of their log₁₀ LRs: an eight factor model used the inputs (four ECG and four clinical) that were most positively associated with diagnosis of ACS (log₁₀ LR above 0.5); a 14 factor model used eight positively and six negatively discriminatory items with log₁₀ LR above 0.5 or below −0.5; a 25 factor model used 14 positively and 11 negatively discriminatory items with log₁₀ LR above 0.2 or below −0.2; a 43 factor model used all inputs with the exception of two that were logically redundant. Table 3 shows the performance of these four models on the hospital 1 dataset. Although performance increased with increasing numbers of data items, the difference between the four models was not great. The 25 factor model was selected for further study based on the test datasets.

With the hospital 1 dataset, the 25 factor model diagnosed 166 of 169 STEMI (98.2%) cases, 101 of 105 (96.2%) non-STEMI, and 167 of 192 (87.0%) unstable angina, whereas ACS was diagnosed for only 41 of 258 (15.9%) patients with stable angina and 11 of 529 (2.1%) patients with non-cardiac chest pain. When combined with clinical opinion (see Methods), sensitivity for diagnosis of MI increased from 97.4% to 98.5%, and for diagnosis of ACS it increased from 93.1% to 94.2%. Specificity decreased from 93.4% to 89.8%.

Performance of 25 factor model on test data

The hospital 2 cohort comprised 1268 patients with a mean age of 62.5 years (range 18–92 years); 727 (57.3%) were men. MI was diagnosed in 319 (25.1%), ACS in 543 (42.8%), and non-cardiac cause for chest pain in 357 (28.2%). Figure 1B shows the distribution of log₁₀ LRs for the 45 data items for this cohort. Although the overall distribution is similar to that for hospital 1 data, the order and magnitude of bars on the charts differ significantly. In particular, ST elevation and new Q waves are the most significant data items for this cohort.

The hospital 3 cohort consisted of 626 patients with a mean age of 60.4 years (range 18–91 years); 366 (58.5%) were men. The final diagnosis was MI in 182 (29.1%), ACS in 300 (47.9%), and pain of non-cardiac origin in 186 (29.7%). Figure 1C shows the distribution of log₁₀ LRs for the 45 data inputs. Results also differed significantly from the hospital 1 dataset. The hospital 4 cohort comprised 152 patients with a mean age of 63.7 years (range 26–89 years); 98 (64.5%) were men. The final diagnosis was MI in 57 (37.5%), ACS in 83 (54.6%), and non-cardiac in 30 (19.7%).

Figure 2 shows ROC curves comparing the performance of the 25 factor model on hospital 2–4 cohorts with the hospital 1 cohort. Table 4 shows performance data for the model applied to the three test datasets. The model correctly identified 98.8%, 95.4%, and 97.5% of STEMIs in the hospital 2–4 cohorts, respectively. Corresponding figures for non-STEMIs were 92.9%, 90.2%, and 88.2%, respectively. The model was well calibrated when applied to test data as fig 3 and fig 4 show.

In the three test cohorts combined, there were 336 STEMIs, 327 (97.3%) of which were identified by the model. Sensitivity increased to 99.1% when output from the model was combined with clinical opinion (see Methods). The model identified 204 of 222 (91.9%) non-STEMIs, and sensitivity increased to 94.1% when model output and clinical opinion were combined. Overall, the model correctly identified 818 of 926 (88.3%) patients with ACS, increasing to 90.1% when combined with clinical opinion. The combination of model output and clinical opinion decreased specificity from 88.5% to 83.0% overall.

DISCUSSION

We have derived and tested logistic regression models for diagnosis of ACS based on clinical and ECG data from the time of presentation. A simple eight factor model performed well on training data. Performance increased progressively as more data inputs were added. A 25 factor model was tested independently on data from a further three centres. Differences in the log₁₀ LRs of inputs between datasets from different centres accounted for the slight decline in performance when the model was applied to test data. We have shown how the output of the model could be combined with either clinical opinion or cardiac marker protein data to identify high risk patients with a degree of accuracy that should be acceptable to clinicians. Each year, an estimated six million people in the USA⁸ and three quarters of a million in the UK¹¹ present to emergency departments with chest pain. Statistical models, as described in this study, may be used as part of chest pain management protocols or as decision support tools to improve triage and early management of patients who present with chest pain or suspected ACS.

This study made use of a large and well validated database to derive and test statistical models according to accepted best practice standards. We have not assessed how use of the models in practice would affect the care of patients with suspected ACS. However, by combining model output arithmetically with clinical opinion, we identified nearly all patients with acute MI, as well as a very high proportion of patients judged to have unstable angina. We have used a composite of a number of factors to define ACS. In future work, we will focus on identifying patients who have an infarction, die, or need cardiological intervention in a defined follow up period. We had access only to systematic cardiac marker protein data for a relatively few patients. Future work will combine model output with markers such as cardiac troponins to further improve identification of high risk patients. The ECG data used for this study were derived from the interpretation of the baseline ECG by emergency department staff, although independent review of serial ECGs by a panel of three was used to set the final diagnosis. Errors in interpretation of the baseline ECG by frontline staff may have affected the potential performance of the statistical models.

The logistic regression models described here perform better than our previously described models for diagnosis of MI.¹² Using ACS rather than MI as an outcome variable increased the importance of clinical factors. Although ECG changes are highly discriminatory for high risk patients,³¹ some clinical data items are also of predictive value.^9–11 The clinical factors that were most discriminatory in this study were hypoperfusion, presence of added heart sounds, sweating, nausea or vomiting, pain radiating to the right arm, age of the patient, exacerbation of the pain by breathing or movement, and the presence of chest wall tenderness. Items that were both negatively and positively associated with a diagnosis of ACS were discriminatory. Risk factors such as diabetes, hyperlipidaemia, and hypertension were not highly discriminatory.

The models described in the present study also performed better than algorithms described by other groups, even when applied to test data from other centres.^8,13–15 The ACI-TIPI⁸ model is a simple one combining ECG data with a few clinical inputs. We¹² and Baxt et al¹⁵ reported poor performance with ACI-TIPI. In the major clinical study with ACI-TIPI, the area under the ROC curve for data from 10 centres was only 0.78—much lower than that for models described in this study. Baxt et al^14,15 has recently described logistic regression and ANN models making use of clinical and ECG data inputs similar to the 43 inputs used in the present study. The performance of logistic regression models was inferior to that of ANNs, and both were inferior in performance to the logistic regression models described here. Using ANNs, Baxt et al¹⁵ achieved a sensitivity of 88.1% and specificity of 86.2% for identifying patients with acute cardiac ischaemia. The algorithm was, however, tested only on the training dataset. The 25 factor model used in this study was 97% sensitive for STEMI, was 92% sensitive for non-STEMI, and identified the majority of patients with unstable angina. We combined model output with clinical opinion to show how it can perform in practice. Although only a theoretical exercise, the combination improved sensitivity for identification of patients with ACS.

The model described in this study performed reasonably but with reduced accuracy when applied to unseen data from other centres compared with training data. Our analysis of log₁₀ LRs shows that individual predictive factors vary between datasets. It seems unlikely, therefore, that such a simple statistical model derived in one centre can ever perform entirely as well when used in different settings. Population demographics, the relation between clinical factors and diagnosis of heart disease, referral patterns, and diagnosis and management of chest pain in receiving departments may differ. The method described here would allow simple models to be locally derived and validated and their simplicity would increase their acceptance by clinical users. Calibration of predictive models for ACS has been considered in only one study, that of Selker et al,⁸ who used ACS-TIPI. Like ACS-TIPI, the model studied here was well calibrated when applied to prospectively collected data. The present study should be regarded as showing the utility of the method. For optimum use of the models described, they should probably be derived again in the settings in which they are going to be used.

Our logistic regression models did not include marker protein data. Such information is difficult to collect systematically for databases of this size. Also, cardiac markers are used to define the final diagnosis and incorporating them into predictive models may lead to prediction bias. Given the well documented short and long term prognostic value of measuring troponins and other marker proteins,^32–35 future studies will need to examine how these measurements are used alongside clinical and ECG data in predictive models. We have previously shown how myoglobin measurements may be used alongside a neural network model.¹⁸ Combination of troponin I measurement with the Goldman algorithm did not improve its predictive ability in a recent study.²⁴ Baxt et al^14,15 used marker protein data in their recently described ANN models. Unfortunately, marker data were missing from many patients and a bias in relation to which patients had marker measurements was possible. Although our models did not incorporate marker protein data, the predictions from the models were very good. Cardiac markers take some hours after the onset of symptoms to become positive. The model based diagnosis in this study was highly predictive of the ultimate diagnosis that included information from troponin measurements. Strategies and protocols for the hours after presentation, including serial measurement of marker proteins, have been developed^1,22 and these have led to development of chest pain centres.³⁶ These centres are an important development for patients at high risk of cardiac ischaemia but their indiscriminate use would be costly and effective decision making based on limited data at the time of presentation in the emergency department is still needed.

The proportion of patients inappropriately discharged from emergency departments in this study was low (around 2%). However, there can be serious consequences for such patients and it is a major cause of litigation.⁷ Of equal concern is the number of patients with chest pain inappropriately admitted to wards or high dependency areas. In the original study with the ACI-TIPI,¹³ the admission rate to the coronary care unit for patients without acute ischaemia was reduced by 30% without a concomitant increase in the rate of inappropriate discharge. A subsequent multicentre trial with ACI-TIPI⁸ was associated with a reduction in coronary care unit admissions from 14% to 10% and a reduction in telemetry unit admissions from 39% to 31%. Admissions to coronary care units or telemetry beds also decreased and rates of discharge increased for patients with stable angina. Rates of admission to high dependency beds or inappropriate discharges for patients with ACS did not change. Use of the Goldman decision tree algorithm^16,24 may also be associated with improved triage practices.²² Statistical models derived from clinical and ECG data also have the potential to improve later management of patients with ACS by identifying those who are at highest risk.^37–40

In conclusion, we have used logistic regression with 10-fold cross validation to derive models to identify patients with ACS by using clinical and ECG data from the time of presentation. A 25 factor model had good calibration and discriminatory ability when applied to test data from three centres other than the one from which training data were derived. We have shown that combining the model output with either clinical opinion or cardiac marker protein data from the time of presentation may further improve identification of patients with evolving coronary syndromes.