Article Text
PDF
Abstract
Background Adverse event(AE) detection in healthcare has traditionally relied upon several methods including: patient care documentation review, mortality and morbidity review, voluntary reporting, direct observation and complaint systems. A novel sampling strategy, known as the trigger tool (TT) methodology, has been shown to provide a more robust and valid method of detection. The aim of this research was to develop and assess a TT specific to ground-based Emergency Medical Services, to identify cases with the potential risk for adverse events and harm.
Methods The study was conducted between March and December 2015. A literature review identified 57 potential triggers, which were grouped together by experts using an affinity process. Triggers for other areas of potential AE/harm were additionally considered for inclusion. An interim TT consisting of nine triggers underwent five iterative rounds of derivation tests of 20 random patient care records (n=100) in two emergency medical services. A final eight-item trigger list underwent a large sample (n=9836) assessment of test characteristics.
Results The final eight-item TT consisted of triggers divided amongst four categories: Clinical, Medication, Procedural and Return-Call. The TT demonstrated an AE identification rate of 41.5% (sensitivity 79.8% (95% CI, 69.9% to 87.6%); specificity 58.5% (95% CI, 52% to 64.8%)). When identifying potential risk for harm, the TT demonstrated a harm identification rate of 19.3% (sensitivity 97.1% (95% CI, 84.7% to 99.9%); specificity 53.5% (95% CI, 47.7% to 59.3%)).
Discussion The Emergency Medical Services Trigger Tool (EMSTT) may be used as a sampling strategy similar to the Global Trigger Tool, to identify and measure AE and harm over time, and monitor the success of improvement initiatives within the Emergency Medical Services setting.
- Emergency Medical Service
- Patient Safety
- Quality
- Adverse Event
- Harm
- Ambulance
- Paramedic
Statistics from Altmetric.com
Key messages
What is already known on this subject?
A trigger tool (TT) is a retrospective sampling strategy used to identify cases with the potential risk for adverse events (AEs) and harm.
TTs offer a novel alternative to conventional methods of AE detection and have been developed and successfully employed among a variety of fields within healthcare, including general healthcare, surgical care, primary care, intensive care, paediatric care as well as pharmacy and laboratory services.
There are no peer-reviewed, published research papers on TTs specific to the general/ ground-based emergency medical services setting.
What this study adds?
Our study employed a mixed methods approach towards the development of a TT specifically for use in the emergency medical services setting.
The Emergency Medical Services Trigger Tool (EMSTT) demonstrated a high sensitivity and moderate specificity in sampling cases with the potential risk for both AEs and harm, and a greater accuracy in identifying such cases compared with a more traditional sampling strategy for AE and harm audit case selection.
Introduction
Adverse event (AE) detection within healthcare has traditionally relied upon several methods, including: patient care documentation review, mortality and morbidity review, voluntary reporting systems, direct observation and complaint systems.1–4 While these methods may be effective for specific patients or circumstances, there is little evidence to suggest they provide a comprehensive or robust system of general AE and harm detection.1–4
A novel approach known as the trigger tool (TT) methodology has, however, seen considerable success as an alternative.5 6 The TT methodology is the application of a retrospective sampling framework that allows for the detection and targeted identification of specific cases at greatest risk for potential AE (unintended consequences associated with medical care) and harm (injury or illness resulting from or contributed to by such occurences). This is accomplished through the recognition of abnormal or unexpected values, measurements, notes or ‘rules’ for any given medical record. The aim of the TT methodology is to evaluate a defined sample of patients to determine whether or not AE and harm are present, and to measure the rates over time as improvement work focuses on the reduction of such events.7 Use of the TT methodology is reported to provide a more time-effective, cost-effective and sensitive means of identifying AEs and harm when compared with traditional methods, such as conventional chart review or voluntary reporting.6 8 Given the success of this approach, TTs have been developed for a variety of fields within healthcare, including general healthcare,7 surgical care,9 primary care,10 intensive care,11 paediatric care12 as well as pharmacy13 and laboratory services.14
Internationally, Emergency Medical Services (EMS) are primarily responsible for the prehospital treatment and transportation of the sick or injured. These services are frequently provided against the backdrop of challenging environments, often with limited resources, and for patients of varying acuity. As a result, the potential for AEs and harm to occur is significant. Despite this, there is a paucity of literature regarding AE detection and reporting within the EMS environment.15–17 Recent efforts to describe systems of AE classification and performance monitoring specific to the EMS context have, however, shown promise.18–20 Furthermore, early prevalidation research aimed at exploring the use of the TT methodology within the EMS environment has also shown considerable success.21
Given the published benefits of the TT methodology and the recent drive towards improving the quality of care and patient safety in EMS, the aim of this research was to develop and test a TT to identify AEs and harm within the EMS setting.
Methodology
Development phase I: literature review
A basic, non-systematic search of the Medline database was conducted using the search terms trigger tool, patient harm, adverse event, emergency medical service and ambulance service to identify peer-reviewed literature regarding the application of the TT methodology in EMS. In addition, a Google web search was conducted to identify examples of EMS-specific triggers or the development of a trigger-based approach for EMS audit case selection. Finally, key experts in patient safety and those specifically with a history in TT development, were consulted to assist in the formulation of triggers suitable for AE detection in the EMS environment.
The Medline search revealed a single peer-reviewed paper that applied a TT-like approach within the EMS setting.21 While rigorously and extensively developed, the tool was limited to a small subset of EMS: Air Medical Services. Based on the results of the Google search, evidence of the use of a trigger-based approach by EMS providers internationally was found within the USA (n=2),22 Wales (n=1)23 and Scotland (n=1).24 The number of triggers identified within each of these examples ranged from 6 to 28. Three examples were applied by their respective organisations as a manual audit and one set was used as a filter of electronic medical record data.
Development phase II: affinity process
A list of the triggers identified in the literature review and web search was compiled (n=57). The authors used an affinity process to refine the trigger list. Similar in concept to thematic analysis, an affinity process is a commonly employed process improvement method that organises large amounts of qualitative-based data into natural and intuitive relationships among the text, through expert consensus.25
During the first step of the affinity process, similar triggers were identified and grouped together. Second, triggers that formed part of an evidence-based clinical pathway (eg, acute coronary syndromes) and triggers aimed at high risk/low frequency cases (eg, rapid sequence intubation) were excluded. Audit in these cases is, generally, carried out as part of a formalised governance framework through the application of quality indicators.19 In addition, triggers involving protocol compliance, or those with no documented correlation to harm were identified and removed. Finally, areas where potential AEs or harm existed, but for which triggers did not exist, were additionally considered for inclusion. An example of this included patients who did not meet evidence-based criteria for spinal movement restriction, yet were immobilised. Following the affinity process, an interim list of nine triggers remained, divided among four categories: Physiological deterioration, Medication issues, EMS response and Restraint and immobilisation (figure 1).
Development of the EMSTT. EMS, emergency medical services; EMSTT, emergency medical services trigger tool.
Development phase III: derivation testing
The nine-item TT underwent small-sample derivation testing in two EMS organisations for further refinement: Hamad Medical Corporation Ambulance Service (HMCAS) (Doha, Qatar) and Mecklenburg EMS Agency (Charlotte, North Carolina, USA) (table 1). A total of five rounds of testing were conducted: three in Doha and two in Charlotte. The testing followed an iterative process to allow for constant refinement and reapplication of the trigger items throughout each round. Consistent with the sampling strategy of the Global Trigger Tool (GTT),7 20 patient care records were identified (by random number table) and selected per round. Each record was then manually reviewed by two reviewers for the presence of one or more of the nine triggers. When identified in the records, triggers were noted, as was demographical and case characteristic data.
Derivation testing – triggers identified in five 20-sample case reviews
The five 20-sample case reviews (n=100) resulted in the identification of triggers in 23 records (23%) (table 1). The presence of triggers ranged from 0 (0%) to 10 (50%) triggers per review round. The most common triggers identified included a return call to the same patient within 24 hours (if not transported with the initial call) (n=8 (8%)) and the administration of an incorrect medication dose (n=7 (7%)). The derivation testing provided several observations, resulting in the refinement of the TT and its application, including:
Deterioration in patient haemodynamic or mental status triggers were merged to a change in early warning score of one point or more from baseline.
Inter-facility and non-emergency transports were added to the exclusion criteria.
Triggers were difficult to assess when certain patient information was absent, or the documentation was unclear or illegible and as such were additionally added to the exclusion criteria (eg, pain scales).
At the end of the derivation testing phase, following further review and refinement through consensus between the authors and expert panel, a final list of eight triggers was produced, divided among four revised trigger categories: Clinical, Medication, Procedural and Return-Call (figure 1).
Development phase IV: performance assessment/test characteristics
Following the derivation testing, the final eight-item Emergency Medical Services TT (EMSTT) underwent a performance assessment at HMCAS (figure 2). The aim of this was to compare the EMSTT as a retrospective sampling strategy for the targeted identification of cases at risk for potential AEs and harm, against a traditional sampling strategy for AE audit case identification, namely random selection.
Emergency medical services trigger tool (EMSTT) performance assessment.
To complete this, all patients serviced by the Emergency Services division of HMCAS for the month of September 2014 were considered for inclusion in the assessment. Patient care records reviewed as part of a key clinical care pathway (eg, ST-elevation myocardial infarction); high-risk procedures (eg, surgical airway); the administration of high-risk medication (eg, paralytic agents); infrequent procedures (eg, intraosseous cannulation) and high acuity inter-facility transport records were excluded from the analysis (figure 2). These high risk/low frequency cases were purposefully removed given that they are, generally, included as part of a 100% audit framework, as a result of their high potential for AEs and harm, a strategy employed by HMCAS. The remaining low risk/high frequency cases are often neglected from audit, and as such, are the target population for the EMSTT.
The records meeting inclusion criteria were identified via the HMCAS centralised electronic database (Microsoft Access 2010; Microsoft, Redmond, Washington, USA). For the EMSTT group, the tool was then applied as a filter query to identify those cases with the presence of one or more triggers (figure 2). For the random selection group, a sample for review was selected via random number table from the cases that met inclusion criteria, to match the sample size of the final EMSTT sample reviewed (figure 2).
As with the GTT, each record review was carried out by a team of two primary reviewers in conjunction for all cases, for both groups. A third reviewer was available to resolve issues where consensus could not be reached. Both primary reviewers were critical care paramedics employed by HMCAS, who were, additionally, involved in clinical governance within the service. Given the focus on development of the tool in this study, and consistent with the strategy outlined by the IHI for the GTT, inter-rater reliability was not recorded, as the primary review team worked in conjunction to come to consensus during the record review and AE/harm classification. All data were captured on a standardised spreadsheet template (Microsoft Excel 2010) for later analysis. See online supplementary data for case review examples.
Records with the presence of one or more triggers were further classified for the presence of AEs and harm. For the purpose of this study, the definition of AE used by the IHI for their GTT was employed, itself an expansion on the WHO Collaborating Centres for International Drug Monitoring’s definition: An event or incident which is ‘noxious and unintended and occurs at doses used in man for prophylaxis, diagnosis, therapy, or modification of physiologic functions’.7 As with the GTT, the definition used in this study was expanded beyond medications to include ‘medical care’ in general. Similarly, the GTTs definition for harm was used in its identification and includes: the ‘unintended physical injury resulting from or contributed to by medical care that requires additional monitoring, treatment or hospitalisation, or that results in death’.7
Given the lack of a widely accepted method for categorising AEs in EMS, the components of the National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) classification system were followed and combined into a simplified classification system for the purpose of this study, namely No AE present; AE present – No evidence of harm; AE present – evidence of harm inconclusive and AE present – evidence of harm. The ‘AE present – evidence of harm inconclusive’ category was included in the classification for those records in which an AE was present, yet it was unclear if harm had occurred given the short duration of care rendered by the ambulance service. For reporting purposes, this category was included with ‘AE present with evidence of harm’ when calculating the harm rate.
For the performance assessment; sensitivities, specificities, positive predictive values, negative predictive values and ORs were calculated for the EMSTT group as a targeted sampling strategy in identifying cases with the potential for AEs and harm, using the random selection group as a control group/reference standard (Minitab 17 2010; Minitab, State College, Pennsylvania, USA). Each group was assessed for cases with isolated AEs, and for cases with AEs associated with harm (and potential harm). In addition, AE rate per 1000 patient encounters and harm rate per 1000 patient encounters were calculated for each group to quantify and compare the ratio of AEs and harm that occurred within each group, into a measurable indicator.
Institutional review board
Ethical approval to conduct the study was granted by the Academic Health System of the Hamad Medical Corporation, Qatar.
Results
A total of 10 187 patient care records were generated in September 2014 and met criteria for AE/harm classification and performance assessment (figure 2). Of these, 351 records (3.4%) met criteria for exclusion and were removed prior to case selection, leaving a total of 9836 records (96.6%).
The electronic filter using the EMSTT returned 993 records (10.1%) with the presence of one or more triggers (figure 2). The Clinical Trigger category made up the significant majority of records (n=965), with the Change in Systolic Blood Pressure Greater Than 20% trigger comprising the single greatest number of triggers within this category (n=925). As a result, the records for this specific trigger were further sampled prior to AE/harm classification and performance assessment. A sample size for estimation calculation was performed using a confidence level of 0.95, CI of 0.1 and estimated proportion of 0.5 to generate a sample size of 97 records (Minitab 17 2010). An additional 10% was added to account for potential missing or illegible records. A sample of 106 records of the Change in Systolic Blood Pressure Greater Than 20% trigger was selected by random number table and included for the AE/harm classification and performance assessment (103 records were included in the final analysis, with 3 records excluded due to missing, incomplete or illegible data).
All records of the remaining Clinical Triggers category (n=40), as well as all records of the Medication Triggers, Procedural Triggers or Return-Call Triggers (n=28) were included for the AE/harm classification and performance assessment. Thus, of the 993 records returned by the EMSTT filter, a final sample of 171 patient care records was extracted for manual review to make up the EMSTT group.
Among this group, the majority of records included the presence of Clinical Triggers (n=143 (83.6%)), followed by Procedural Triggers (n=19 (11.1%)), Return-Call Triggers (n=8 (0.81%)) and Medication Triggers (n=1 (0.1%)) (figure 2). Among the individual TT items the trigger Change in Systolic BP >20% from First Measurement (n=103 (60.2%)) accounted for the greatest number of records with evidence of both AE (n=30 (17.5%)) and harm (n=16 (9.4%)). Six records included the presence of multiple Clinical Triggers. There were no records that included the presence of more than one trigger category.
For the random selection group, a sample was selected via random number table to match the sample size of the final EMSTT group reviewed ((n=159 (12 missing or illegible records excluded)). Table 2 outlines the AE and harm classification results. As a sampling strategy for AE and harm audit case selection, the EMSTT demonstrated greater accuracy in identifying cases with the potential risk for AEs and harm, when compared with random selection (EMSTT AE rate–7.22/1000 patient encounters, EMSTT harm rate–3.36/1000 patient encounters; random selection AE rate–1.83/1000 patient encounters, random selection harm rate–0.1/1000 patient encounters).
AE and harm classification
When used to identify records at potential risk for AE, the EMSTT demonstrated a sensitivity of 79.8% (95% CI, 69.9% to 87.6%) and a specificity of 58.5% (95% CI, 52% to 64.8%) (OR 5.6 (95% CI, 3.1 to 9.9)) (table 3). When identifying records at potential risk for harm, the EMSTT demonstrated a sensitivity of 97.1% (95% CI, 84.7% to 99.9%) and a specificity of 53.5% (95% CI, 47.7% to 59.3%) (OR 37.8 (95% CI, 5.1 to 279.9)). Table 4 gives a breakdown of the basic demographical data of the EMSTT and random selection groups.
EMSTT AE and harm analysis
Demographical data
Discussion
AE detection has become an essential component of patient safety and clinical governance within healthcare.27 28 The importance of understanding this burden is poised to become even more significant considering recent estimates that place death due to medical error as high as the third leading cause of mortality in the US alone.29
There are a multitude of systems and methodologies that currently exist with the aim of identifying AEs and harm, at various stages in the process of care of a patient. Use of triggers is one such approach that has seen considerable success as a broad, overarching sampling strategy towards identifying base AE and harm rates, within a number of healthcare settings.
By comparison, the call to develop similar such systems within EMS has been relatively slow.15 16 Much of the focus of patient safety and care quality has been on the development of quality auditing or measurement systems within specific clinical care pathways. Quality measurement undoubtedly has a significant role to play in monitoring EMS performance and reliability; however, it does not provide significant insight into potential AE and harm detection. Audit, by design, is not principally focused on identifying AEs or harm, but instead is a tool used to focus on deviation from accepted standards of care. While it could be argued that in doing so, quality indicators provide a secondary role in identifying AEs and harm, they nonetheless remain largely confined to specific care pathways. Furthermore, they do not offer one of the primary benefits for which the TTM is designed–a focused, purposeful sampling strategy.
The development of EMSTT followed a similar approach outlined by the development of the GTT, as with many TTs developed for other branches of healthcare. Outside of this, however, comparison with these hospital-based TTs is difficult, given the variation in focus on patient populations and areas of service. A literature search revealed only a single peer-reviewed research article that adopted a trigger-like approach towards identifying AEs in the EMS environment. However, as highlighted previously, this was restricted to a small subset of EMS, namely Air Medical Services, making the comparison of their results with this study difficult as well. In addition, the tool developed in the cited study was limited to the development and content validity, and was not applied to a sample as part of a performance assessment, further reducing comparability.21
One of the most significant observations to emerge from this study was the limited research into EMS specific systems of AE classification. As comprehensive as the NCCs MERP classification system is, its in-hospital focus has made it difficult to extrapolate to the EMS environment, a factor which is further confounded by the limited time in which patients are exposed to EMS care. A single peer-reviewed study does exist that attempts to outline a classification system specific to EMS; however, was limited in its development, and as such, was not applied in this trial.20
Limitations
Evaluation of the EMSTT excluded several records from the filtering process, including evidence-based care pathways and high-risk/infrequent medications and procedures. Although these cases were seemingly small in proportion, the potential exists that their exclusion skewed the results of the analysis. The removal of these records was deliberate, given the alternate audit programs that exist for these specific cases within the participating service and in many services internationally. This limitation represents an avenue for further research and evaluation of the EMSTT in a sample where such exclusion criteria have not been applied.
A significant number of records in our study included the presence of the trigger Change in Systolic Blood Pressure Greater Than 20%, which resulted in the further sampling of this trigger due to the proportion of its occurrence. Reviewing a sample of the records with this trigger could have potentially limited the results by not allowing consistent review of every case. As a result, further longitudinal study may be necessary to determine the scale of effect such an additional sampling strategy may have on long-term use and reporting with the EMSTT.
Finally, it could be argued that the relatively low specificity of the EMSTT has the potential to produce a significant number of false negatives. This, in turn, has the potential to skew the results when reporting on AE and harm rates, and more importantly when identifying critical areas for improvement based on these results. Again, longitudinal study of the application of the tool in multiple settings is necessary to determine the true impact of these factors.
Conclusions
TTs are a successful strategy for identifying the prevalence of AE and harm within healthcare. This study represents the first effort to develop and test an EMS-specific TT, with the aim of identifying patients at risk for AE and harm. This study shows that the EMSTT incorporates several desirable traits of the ideal TT and represents an avenue to improve an EMS agency’s ability to identify AE and harm, and track improvement over time.
References
Footnotes
Contributors DMW, KSM, RCO and LAHAS conceived the idea for the research. ILH, JMB, DMW, KSM and LAHAS contributed towards data collection and analysis. ILH was primarily responsible for the draft of manuscript. All authors contributed towards the final manuscript for submission.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.