The effect of a Picture Archiving and Communications System (PACS) on diagnostic performance in the accident and emergency department
- 1Health Economics Research Group, Brunel University, Uxbridge, Middlesex UB8 3PH
- 2Health Services Management Centre, University of Birmingham, Birmingham
- 3Accident and Emergency Department, Hammersmith Hospital, London
- 4A&E Medicine Academic Unit, Cancer Centre, Prince of Wales Hospital, Shatin, New Territories, Hong Kong
- Correspondence to: Gwyn Weatherburn (e-mail: )
- Accepted 10 November 1999
Objective—A study has been conducted to identify the benefits to the accident and emergency (A&E) department of a hospital wide Picture Archiving and Communications System (PACS).
Methods—The study was conducted in two parts: firstly while the hospital was using conventional radiographic films, and secondly when the PACS was in operation. For each part of the study, the diagnoses of radiographic images made by A&E clinicians were compared with those made by radiologists. This resulted in the estimation of the incidence of false negative findings by the A&E staff. The management of patients with such findings was studied to identify those for whom a change of treatment was required. Such data for the two periods, when film and when PACS was used, were compared.
Results—It was found that the overall rate of misdiagnoses across all A&E patients who had radiography was low in both periods and there was a significant reduction when PACS was used (1.5% for film and 0.7% for PACS, 95% CI for difference between proportions: −0.014 to −0.0034), but the rate of serious misdiagnoses involving patient recall did not change significantly (95% CI for difference between proportions: −0.0059 to +0.0001).
Conclusions—When PACS was used the diagnostic performance by A&E staff improved by reducing false negative interpretations but the rate of serious misdiagnosis did not change.
In the accident and emergency (A&E) department radiographic images are frequently used to assist in the initial diagnosis and management of the patient. Often A&E clinicians are unable to obtain immediate radiological support and so have to rely on their own interpretation of the images to assist in immediate treatment decisions. Junior medical staff working in an A&E department have very limited radiological training or experience but may have to interpret images without access to a radiologist's report for 16 hours per day from Monday to Friday, and 24 hours per day over weekends. Misdiagnosis in the A&E department is an apparently common and potentially serious problem, and previous studies have found that misdiagnosis rates by A&E staff range from 0.6% to 7%.1–13 Failure to interpret radiographic images correctly is a common cause of diagnostic error and litigation in the A&E department.6, 14
Radiology departments have begun to make use of new technology known as Picture Archiving and Communications System (PACS) that acquires, transports and stores radiographic images electronically. The number of PACS being installed in hospital settings continues to expand rapidly.15 A&E departments are among those most commonly connected to the PACS network as the clinicians in this specialty require rapid access to radiographic images to support decisions about patient management.
The Hammersmith Hospital in West London has a General Electric hospital wide PACS and, with the exception of dental images, the whole hospital is “filmless”. Thus A&E clinicians routinely view soft copy radiographic images on work stations within the A&E department almost immediately after they have been taken, and use these to aid rapid decisions about the treatment of the patient's condition. Various tools are available to the clinicians to allow them to manipulate the soft copy images, including variation of the grey scale and contrast, and zooming to magnify part of the image and increase its resolution. PACS might be expected to be beneficial to A&E clinicians by enabling them to manipulate the soft copy images and, thus, potentially improve their diagnostic performance. To assess whether this benefit was realised at Hammersmith, a case study was undertaken that monitored the diagnostic performance of A&E clinicians before and after the PACS implementation. This study was part of the independent evaluation of the Hammersmith PACS project that was conducted by the Health Economics Research Group (HERG) on behalf of the Department of Health.16
The hypothesis tested in this study was that, compared with the use of film images, PACS would reduce the number of misdiagnoses by A&E clinicians and consequently the number of patients who had to be recalled to the hospital for reassessment. Thus, the aim of this study was to assess the impact of PACS on the incidence, and consequences, of misdiagnoses in the A&E department at the Hammersmith Hospital. Data were collected over two periods: when conventional film images were used and when PACS was in use. Data were collected on all images reported as positive (that is, abnormality present) by the radiologist but seen as negative (that is, no abnormality present) by the A&E clinician. As a matter of routine audit, a senior A&E clinician compared each relevant radiologist's report of an abnormality with the findings of the A&E Senior House Officer (SHO). If it appeared that a diagnostic error had been made by the A&E clinician, a decision was made about whether treatment should be changed. If a change was thought necessary then the urgency of such a change was defined in terms of whether the patient should be recalled for further treatment. By this method potential false negative findings on radiographic images by A&E clinicians were identified. These cases were classified by the seriousness of the condition and any action recommended. Table 1 gives the classification. The routine audit procedure in operation at the Hammersmith did not monitor all negative radiographic findings by the radiologists. Ideally, false positive findings would also be investigated. However, an investigation of false positive findings could not be undertaken because it was not feasible for the auditing clinicians to check all positive A&E reports with the radiology reports for all patients over two six month periods in addition to their normal workload. Thus, the term “misdiagnosis” is used in this paper to refer to false negative cases only, where the gold standard is taken as the radiologist's report.
Data based on conventional film images were collected from 31 March 1992 to 30 September 1992. The Hammersmith Hospital became “filmless” at the end of March 1996 and the post-PACS data collection period was from 1 April 1996 to 30 September 1996. The same seasonal timing of the “before” and “after” periods was designed to reduce the likelihood that the mix of patients would be different for the two periods, and ensured that the junior A&E staff (who work in A&E for six months starting either at the beginning of February or August) were of the same experience in both studies. By the time of the PACS component of the study, both the A&E and the radiology departments had been using PACS images for more than a year.
Data on the total number of A&E patients radiographed during each study period were obtained from the Radiology Information System (RIS). The details of every radiological examination were routinely recorded on RIS either by the radiographer or by the X-Ray receptionist. It should be noted that some patients had more than one radiological examination and, thus, the total number of examinations was greater than the total number of patients radiographed. Data on the number of patients attending the A&E department during each study period were obtained from the local A&E Register. These data included both new and follow up attenders.
The two rounds of data collection were initially compared in terms of the number of patients presenting with a new problem and the number of follow up attenders. Data were also available on the number of A&E attenders who were radiographed, and the body area of the radiological examination. The data from each round were compared by calculating differences in proportions of patients and 95% confidence intervals around the differences.17
The overall misdiagnosis rates for the film and PACS periods were then compared. The rates were first calculated using the total number of A&E attenders as the denominator and then recalculated using the number of A&E attenders who were radiographed as the denominator. The misdiagnosis rates for the film and PACS rounds were then compared separately for adults (defined as being 16 years of age or older) and children (defined as being less than 16 years of age). Again, these comparisons were made by calculating 95% confidence intervals around the differences in proportions. Finally, data concerning the distribution of misdiagnoses by body area are reported.
The results for the comparison of film and PACS periods in terms of patient characteristics are given in tables 2 and 3. The mix of patients presenting at the A&E department was different between the two periods, in two important respects. Firstly, a significantly smaller proportion of patients attending in 1996 were follow up patients and, thus, a significantly larger proportion were presenting with a new problem (table 2). Secondly, a significantly larger proportion of patients attending in 1996 received a radiological examination (table 3). Apart from examinations of an upper limb, for all categories of radiological examinations, a larger proportion of patients received an examination in 1996. This increase was particularly marked for examinations of the chest. In 1992, approximately 6% of all A&E attenders received a chest radiograph, whereas in 1996, approximately 12% of attenders had a chest radiograph.
The results for the overall comparison of the film and PACS data collection periods, in terms of misdiagnosis rates, are shown in table 4. During the film period a total of 39 patients were misdiagnosed when film was being used, giving an overall misdiagnosis rate of 1.5% in those patients who were radiographed. The number of patients who were recalled for review (misdiagnosis categories 1 to 3) was 16. During the PACS period a total of 35 patients were misdiagnosed when PACS was being used, giving an overall misdiagnosis rate of 0.66% in those patients who were radiographed. The number of patients in the PACS period who were recalled (misdiagnosis categories 1 to 3) was 20.
The proportion of misdiagnoses among A&E attenders who were radiographed was statistically significantly lower in the period when PACS was being used compared wih the period when film was used. However, the proportion of serious misdiagnoses among A&E attenders who were radiographed was not significantly different between the two periods. The data were analysed separately for adults and children (tables 5 and 6). For adults, there was a significantly lower proportion of misdiagnoses overall when PACS was used, but the rate of serious misdiagnoses, requiring patient recall, was the same for the two periods. For children, the misdiagnosis rates, both overall and for serious misdiagnoses, were the same for the PACS and film periods.
Tables 7 and 8 and figures 1 and 2 show the distribution and severity of misdiagnoses between body areas, for both adults and children. The results indicate that the misdiagnoses tended to relate, principally, to examinations of the upper or lower limb or skull examinations. There are no pronounced differences in the distributions between the film and PACS periods.
COMPARISON WITH OTHER STUDIES
The number of misdiagnoses by A&E staff identified in this study was very low, for both film and PACS. However, the rates for both film and PACS images identified here are consistent with those reported in other studies.1–13 In addition these results compare well with a study of conventional radiographic films by Walsh-Kelly et al18 who reported that radiograph misinterpretation by emergency department physicians occurs but is unlikely to result in significant morbidity.
This pragmatic study of two periods of normal mixed workload has shown that A&E clinicians on six month rotations were able to use the new technology without detriment to patient care. It is thus in general agreement with the results of the experimental study by Gillard et al19 who compared the interpretation of digital and film images for skeletal fractures and the acceptance of the technology by A&E staff. In this paper we have not considered the issues of user acceptance and training, but refer readers to other publications about the evaluation of the Hammersmith PACS system, which conclude that the technology has been widely and effectively accepted in the hospital.20–22
CONFOUNDING FACTORS THAT OCCURRED OVER THE PERIOD OF THE STUDY
The implementation of the full PACS system occurred over a four year time scale and there were several changes other than the implementation of PACS that may have influenced the results of this study. Inevitably there were changes in personnel in the A&E department but the junior doctors were of the same level of experience in both periods of the study. The induction training for the SHOs changed between the periods: at the time of the conventional film study (1992), there was a four day induction training course, which included integrated tuition in the use of imaging. By 1996, training was provided on a two day generic A&E course outside the hospital, with specific training in the use of the PACS system held in house. In addition, by the time of the PACS period there were more “middle grade” doctors available to provide advice during normal working hours, but there had been a reduction in consultant presence. Outside normal working hours, including weekends, there was no change in the grade of doctor available.
For the first data collection period, patients were radiographed in the general radiology department on the floor immediately above A&E. In 1993, a dedicated radiology room for A&E patients was built within the A&E department, which was supervised by a superintendent radiographer with specific responsibility for A&E work. The close proximity of a dedicated A&E radiology room is one possible reason for the increase in requests for radiological examinations during the PACS period. The finding of an increase in the use of radiological examinations can be interpreted either as a change in the characteristics of patients seen in the A&E department or as a change in the behaviour of A&E clinicians for other reasons, such as an increased fear of litigation. There may be a link between the larger proportion of patients receiving a radiological examination in 1996 and the larger proportion of patients presenting with a new problem.
During the film period, the A&E films were “hot reported” during normal working hours but during the PACS study the images were not routinely reported while the patient was in the A&E department. In 1996 the A&E staff made anecdotal comments that the written radiologists' reports took longer to arrive than in the past. These two factors may have caused the A&E staff to rely less on radiologists and instead seek the opinions of other senior clinicians or the A&E radiographers who were located nearer than the radiologists. This process of more frequent consultations may have helped to reduce the number of mistakes that were made by inexperienced junior A&E staff. Balancing this effect, PACS provided the potential for images to be viewed simultaneously in A&E and Radiology. It is possible that this may have led to the A&E doctors being more willing to consult radiologists by phone for an opinion, as this would not necessitate a visit to the radiology department. The consequence of this may have been the observed reduction in the number of images misdiagnosed, although we have no data on the number of telephone consultations.
ACCURACY OF RIS DATA
The accuracy of the RIS information on the numbers of patients radiographed must be treated with caution. It was found in this study that on some occasions a patient's details appeared not to have been entered on RIS even though the patient had been radiographed and the images were available. Thus, the study data on the number of patients radiographed may be an underestimate and so the misdiagnosis rates may be inflated. This is a known inaccuracy in the data that could not be controlled by the researchers. However, there is no reason to believe that such errors would have been made more often in one part of the study than the other.
OTHER RECENT TECHNOLOGICAL DEVELOPMENTS OF PACS
Further technical developments, such as automated audit systems23 and teleradiology,24–26 may increase the benefits of PACS. Automated audit can speed up the audit process and allow any necessary changes in treatment to be made earlier. Teleradiology can provide ready access to a radiologist for “hot” or on line reporting of all radiographic images thus reducing the possibility of misdiagnosis of the images by A&E staff.
Gwyn Weatherburn participated in the formulation of the protocol and design of the study. She set up and coordinated the data collection, and undertook analysis and interpretation of the results and the writing of the paper. Stirling Bryan initiated and coordinated the formulation of the study. He participated in the analysis of the data and the interpretation of the results and the writing of the paper. Anne Nicholas was involved in the initial design of the protocol. She audited the radiographic reports, collected and documented data and organised the follow up of patients in both phases of the study. She advised on and approved the final version of the paper. Robert Cocks participated in the initial design of the study protocol and in the clinical follow up of radiological findings, audit and data collection for the first part of the study. He participated in the writing of the paper, approved the final version and is guarantor for the study.
The authors wish to thank Professor Martin Buxton at HERG and the staff in A&E at Hammersmith Hospital for their assistance with this study, the anonymous referees for their very helpful comments, and Karen Arnold for her secretarial support.
Funding: this study was funded by a grant from the Policy Research Programme of the Department of Health.
Conflicts of interest: none.