Objective The learning curve–cumulative summation (LC-CUSUM) test allows for quantitative and individual assessments of the learning process. In this study, we evaluated the process of skill acquisition for performing endotracheal intubation (ETI) in three emergency medicine (EM) residents over a 2 year period in their first 2 years of their EM residency.
Methods We evaluated 342 ETI cases performed by three EM residents using the LC-CUSUM test according to their rate of success or failure of ETI. A 90% success rate (SR) was chosen to define adequate performance and an SR of 80% was considered inadequate. After the learning phase, the standard CUSUM test was applied to ensure that performance was maintained.
Results The mean number of ETI cases required to reach the predefined level of performance was 74.7 (95% CI 62.0 to 87.3). CUSUM tests confirmed that performance was maintained after the learning phase.
Conclusions By using the LC-CUSUM test, we were able to quantitatively monitor the acquisition of the skill of ETI by EM residents. The LC-CUSUM could be useful for monitoring the learning process for the training of airway management in the practice of EM.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
While the Accreditation Council for Graduate Medical Education recommends that emergency medicine (EM) residents perform at least 35 endotracheal intubations (ETIs) prior to graduation,1 there is little evidence regarding the training required to successfully acquire ETI skills in EM. Several studies have made recommendations regarding the number of cases needed to learn ETI skills for different medical specialties. For example, it is recommended that anaesthesia residents perform 27–57 ETIs to reach 90% success.2–4 Student paramedics required exposure to more than 15–25 live ETI encounters across a range of clinical settings to achieve success rates greater than 90%.5 In the emergency department (ED) setting, the patient population for airway management includes critically ill and injured patients with some degree of instability.6 EM residents may have very different learning experiences given the different patient populations, clinical techniques and teaching conditions they encounter.
Performing a recommended number of procedures under supervision is the standard method for procedure training in EM residents but there is a lack of objective evidence about individual competency. Recently, application of statistical methods have been used to analyse the learning curve (LC) of surgical and interventional procedures. Biau et al7 reviewed the literature from 1991 to 2006 and noted 22 papers that used the cumulative summation (CUSUM) test to determine the LC. Biau et al8 introduced a modification of the CUSUM method, called the LC-cumulative summation (LC-CUSUM) test, that was developed to signal when an individual’s performance has achieved a predefined level of competence. This new method has been applied to the analysis of the LC and quality control for monitoring clinical performance.9–11 The LC-CUSUM test, to the best of our knowledge, has not yet been well represented in the EM literature. The goal of this study was to evaluate the process of skill acquisition for performing ETI in three EM residents in their first 2 years of EM residency using the LC-CUSUM test. The primary outcome was the number of ETI cases required to pass the LC-CUSUM test.
Materials and methods
The study was retrospective using prospectively collected data over a 24 month period from an airway management registry of two EDs. The institutional review board of the institution approved the study.
Study setting and population
The study was conducted at two tertiary urban university hospitals with annual ED visits of approximately 35 000 patients. The two EDs run 4 year EM residency (PGY 2–5) training programmes jointly, and accept three residents per year. Intubations are usually performed by attending faculty emergency physicians or EM residents. When the first year EM residents were allowed to intubate, the devices and methods used were chosen by supervising faculty members or senior residents.
We reviewed all 1191 registry forms completed between March 2008 and February 2010, and found 342 ETI cases that had been attempted by all of the three EM residents who started their first year of EM residency in March 2008. All of the residents had completed 1 year internships before starting their EM residencies, and had little prior airway management experience (table 1). They received formal training on ETIs by an attending faculty emergency physician. They also scheduled a rotation in anaesthesia for at least 3 weeks before the start of their EM residency. The exclusion criteria included ETI cases that were intubated by means other than direct or video laryngoscopy, such as use of an elastic bougie, retrograde intubation, cricothyroidotomy, tracheostomy or nasotracheal intubation. In fact, in all of the 342 ETI cases that had been attempted by the three EM residents, intubated was achieved using the technique of direct or video laryngoscopic ETI. The successes or failures of the 342 ETI cases were included in the LC-CUSUM test.
Methods and measurements
Registry data were written on a standard single page data sheet for all ED patients that had been intubated by the physician who performed the intubation. During each morning round, the site investigator reviewed completed forms to prevent data omission. The main data collected included patient demographics, assessment of difficulty, intubation devices and methods used in each attempt, success or failure of each attempt, and immediate complications. ETI success was defined as the placement of an endotracheal tube within two attempts without immediate complications, such as oesophageal intubation or dental trauma. Successful intubation was confirmed by chest movement, auscultation and capnography. If intubation failed after two attempts, the faculty members or supervising senior residents took over the procedure and secured the airway. An ETI attempt was defined as a single pass of a blade into the mouth.
The LC-CUSUM test was developed to signal when performance was sufficiently far away from an inadequate performance level to consider the performance as adequate.8 It assumes that the process being studied is not being performed successfully (ie, the trainee is not proficient) at the start of monitoring and signals when the process can be considered to be competently performed. LC-CUSUM scores (Slc) were calculated from the successive outcomes and usually plotted below the x axis. The score Slc will fall with an ETI success and rise with an ETI failure. When Slc reaches a certain predefined limit, the LC-CUSUM test is said to emit an alarm indicating the performance has reached an adequate level.12 The LC-CUSUM test has a holding barrier of zero, which prevents the score from drifting too far away from the decision limit. When the graph hits this limit, it stays at zero and monitoring continues. In this way, the trainee will not have to compensate for all the accumulated failures to show acceptable performance.9
In comparison, the CUSUM is designed to detect a shift from an adequate to an inadequate performance level. The successive outcome is recorded above the x axis. The CUSUM score (Sc) increases with an ETI failure and decreases with an ETI success. If the score Sc crosses the predefined limit, the CUSUM test is said to emit an alarm demonstrating inadequate performance. The CUSUM can be used after the LC-CUSUM has shown that the resident reached proficiency to ensure that the performance is being kept at an acceptable level. We established acceptable and unacceptable success rates as 90% and 80% based on previous studies2–5 of ETI skill acquisition and consensus among EM physicians in the department. For the LC-CUSUM, a limit of h=−3.15 was chosen on the basis of numerical simulations so that the risk of declaring a trainee proficient when his or her performance is inadequate (type I error) was limited to 10.0% over 100 ETI cases, and the risk of not declaring a trainee proficient although his or her performance is adequate (type II error) was limited to 16.2%. For the CUSUM, a limit of h=3.2 was chosen to yield type I and II errors of 9.4% and 10.2%, respectively, in 100 ETI cases. Detailed explanations with regard to calculation and formulation of the LC-CUSUM and CUSUM tests are given in the online supplementary appendix I.
LC-CUSUM calculations were performed using Microsoft Excel 2010 (Microsoft, Washington, USA). Continuous variables are expressed as means with 95% CIs, while categorical variations are expressed in numbers and percentages. A p value of p<0.05 was considered statistically significant.
A total of 342 ETI cases were performed by the three EM residents from March 2008 to February 2010. The mean number of ETI cases over a 2 year period per resident was 114.0 (95% CI 100.5 to 127.5), and the mean success rate of the ETI cases was 89.1% (95% CI 85.0 to 93.1) (table 1). All failed intubations by the residents were subsequently successfully completed by faculty members or senior residents.
The mean number of ETI cases required to pass the LC-CUSUM test was 74.7 (95% CI 62.0 to 87.3). The LC-CUSUM graph for the residents is presented in figure 1. The LC of the residents showed the improvement of ETI skill (figure 2), with an 80% success rate after a mean of 30 cases. However, the pooled success rate did not reach 90% within a mean of 114 cases.
The LC-CUSUM and related statistical process control charts represent a more customised approach to individual LCs than the usual approach, which relies on uniformly applied measures of competency, such as performance of a specific number of procedures. We assessed the learning process of ETI skill for the three EM residents over a 2 year period in their first 2 years of EM residency. In this study, the LC-CUSUM signalled that the residents were competent for the ETI procedure after a mean of 74.7 ETI cases. In studies of emergencies outside of the operating room, whether taking place in the ED or in the intensive care unit, the risk of a difficult intubation was almost twice as likely.13 While rapid sequence intubation is widely used, EM residents may be allowed fewer attempts to achieve successful ETI on unstable patients in the ED than in the operating room. Less chances of attempting ETI might be the reason that the three EM residents needed more ETI cases to reach 90% success.
The LC for most airway devices is biphasic: reasonable competency can be achieved within 30 cases but performance continues to improve even after 100 cases.14 A patient with a difficult airway demands a physician who has the expertise to modify his or her technique as necessary. Without deliberate practice, many practitioners will not advance beyond the stage of automaticity and will perform at mediocre levels for the rest of their careers.15 Individualised monitoring and training might counter arrested development by providing challenges outside a physician’s comfort zone as opportunities for deliberate practice. New standard guidelines for procedure training, such as ETI, would also be necessary and should be targeted to different stages of training. Acceptable failure rates are different not only for different medical specialties but also at different stages of training.16
In the LC-CUSUM, introduced by Biau et al,8 the two initial parameters (acceptable (p0) and unacceptable failure rate (p1)) define the weight value; a weight for success and a weight for failure. An additional variable that influences the outcome of LC-CUSUM tests is the practical limit in number of cases available to each trainee (Nmax).17 If provided with a large enough number of cases, all trainees would eventually be considered competent by chance.18 However, for smaller values of Nmax, in the case of critical care, the true type I error rate, α, tends to increase and the type II error rate, β, tends to decrease. The value of h0 is the fourth parameter, which defines a terminating barrier. Biau et al12 recommended using simulation in order to achieve a false positive rate (type I error) equal to or less than 10% within a given Nmax.
The LC-CUSUM test allowed us to quantitatively monitor competency among EM residents for ETI performance. After individualised assessment has been accomplished, competency based education against fixed achievement standards should be adopted for uniform educational outcomes.19 If the guideline on what is an acceptable success rate was set, we could adapt the LC-CUSUM test for different stages of training in ETI procedures. An easy tool for measuring performance quantitatively would aid the objective evaluation of resident performance, offering better training opportunities. Further studies should be performed with the goal of individualising training of EM residents, with an understanding of the diversity of the LC among EM residents.
Some limitations of our study should be mentioned. First, the LC-CUSUM test was applied to a very limited sample. Thus ED data with different numbers of residents and different training environments might show different results. With this limitation, the LC-CUSUM test is valued as a potentially salient methodology to the task of individualised assessment in the high stakes environments of EM. Second, there are no agreed guidelines for the success rate of adequate and inadequate ETI skills acquisition. Different target rates at different stages of training could produce different results. Third, all outcomes in this study were self-reported and therefore may have been subject to self-reporting bias. Future studies should consider a prospective design with an attending EM physician present. Fourth, we explored a restricted competence in emergency airway management. Other important skills, such as rapid sequence intubation, assessing the environmental and external variables, and advanced intubation techniques, were not evaluated in this study.
We explored the usefulness of applying the LC-CUSUM test, relatively new to EM, to the task of establishing minimum competency criteria for a high risk procedure. We were allowed to quantitatively monitor the acquisition of ETI skill by EM residents using the LC-CUSUM test. The numbers of ETI cases needed to pass the LC-CUSUM test were variable. The LC-CUSUM test could be useful for monitoring a learning process for the training of airway management in the practice of EM.
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online appendix
SJ and YC contributed equally to this study.
Contributors SJ and YC: drafting of the manuscript, acquisition of the data and analysis of the data. HC, BK and TL: critical revision of the manuscript for important intellectual content. HK: study concept and design, interpretation of the data, critical revision of the manuscript for important intellectual content and final approval of the version to be published.
Competing interests None.
Ethics approval The study was approved by the institutional review board of Hanyang University Seoul Hospital.
Provenance and peer review Not commissioned; externally peer reviewed.