Article Text
PDF
Abstract
Objective Analysis of 72-hour ED revisits is a common emergency medicine quality assurance (QA) practice. Our aim was to compare the perceived root cause for 72-hour ED revisits between senior physicians (attendings) and trainees. We proposed that discordance in perception of why the revisit occurred would guide improvements in 72-hour revisits QA and elucidate innovative educational opportunities.
Methods Questionnaire-based observational study conducted in an urban academic paediatric ED. Treating attendings and trainees independently completed questionnaires on revisit cases. The primary outcome was the revisit’s perceived root cause, dichotomised into ‘potential medical deficiency’ or ‘not potential medical deficiency’. Discordance between provider pairs was measured, stratified by revisit disposition.
Results During the study period, 31 630 patients were treated in the ED, 559 returned within 72 hours and 218 met inclusion criteria for paired analysis. The proportion of cases assigned ‘potential medical deficiency’ by the attending and trainee was 13% and 9%, respectively. Discordance in the dichotomised root cause between attendings and trainees was 17% (38/218, 95% CI 12% to 22%). Revisit cases requiring admission revealed attending-trainee discordance of 25% (23/92, 95% CI 16% to 34%).
Conclusions Attendings and trainees frequently disagree on whether a potential medical deficiency was the root cause for an ED revisit, with more disagreement noted for cases requiring admission. These findings support the premise that there may be opportunities to improve 72-hour revisits QA systems through trainee integration. Finally, reuniting attending-trainee pairs around revisit cases may be a novel educational opportunity.
- bounce back
- feedback
- medical education
- patient readmission
- pediatric emergency department
- pediatrics
- physician behavior
- return visits
- revisit
- root cause analysis
- medical deficiency
- quality assurance
- quality improvement
Statistics from Altmetric.com
- bounce back
- feedback
- medical education
- patient readmission
- pediatric emergency department
- pediatrics
- physician behavior
- return visits
- revisit
- root cause analysis
- medical deficiency
- quality assurance
- quality improvement
Key messages
What is already known on this subject?
Review of revisits is a common quality assurance practice in emergency medicine. Including the trainee in the revisit process may lead to an improved process and foster development of the trainee.
What this study adds?
In this prospective study of 218 patient revisits, we found that senior physicians and trainees often disagree about whether or not a potential medical deficiency was the root cause of a 72-hour revisits. This high discordance implies that the trainees’ perspective is important to integrate into a revisit QA system. Furthermore, reuniting colleagues around revisits emerges as a novel educational opportunity.
Introduction
Feedback is central to the learning process.1 In emergency medicine, Croskerry highlighted both the importance of, and impediments to, physicians’ receipt of high-quality feedback.2 In recent years, novel feedback systems have demonstrated their value through notable improvements in patient care, quality metrics and the development of unique educational curricula.3–13
In emergency medicine, review of 72-hour ED revisits is an established quality assurance (QA) practice aiming to provide physician and departmental feedback. Published rates of potential medical deficiencies as the root cause of such visits vary between 1% and 13%.14–19 The reported benefits of return visit reviews include identifying gaps in care and medical errors, but the resource intensive process for a relatively rare event has been also noted.16 18 20
In 2014, Shy et al published a series of recommendations for creating a meaningful 72-hour ED revisits review process. Prominent within their recommendations is encouraging the integration of the trainee as case screeners,21 which is congruent with ongoing trends in medical education.11 12 To our knowledge, no studies have included the trainees’ perspective on reviewing their own revisits.
Our institution has an established 72-hour revisits case review system conducted by the senior physicians (attendings).18 Inspired by Croskerry’s argument regarding the value of, and need for, physician feedback2 and following the recommendation by Shy et al to integrate the trainee’s perspective,21 we expanded our model and share our findings in this report.
Project aim/hypothesis
Our aim was to compare attending and trainee perceived root cause for 72-hour ED revisits. We proposed that a notable discordance in perception of why the revisit occurred would support including trainees in revisit QA systems and potentially uncover novel educational opportunities.
We postulated that trainees may be overcritical of their care and when compared with attendings, more often attribute 72-hour return visits to a potential medical deficiency. Uncovering a directional nature of discordance would further guide development of revisit feedback systems and guide educational initiatives to expand trainees’ learning beyond the initial patient encounter.
Methods
Study design and setting
This prospective, questionnaire-based, observational study was conducted in the ED of an urban, academic, paediatric hospital from August 2015 to February 2016. The ED had 60 000 annual visits with an admission rate of 19.5%.
In our ED, paediatric emergency medicine (PEM) attending physicians in combination with trainees (residents and PEM fellows) treated 53% of the patients with the balance of patients cared for by independent attending physicians. PEM attendings completed both a paediatrics residency and a PEM fellowship while our urgent care paediatric attendings completed a paediatrics residency. Most attending-only care represented urgent care paediatricians who saw the majority of low acuity patients. All patients seen by a trainee were staffed with a PEM attending.
Through review of our administrative databases, the 72-hour revisits rate for 2015 was 3.1%; 29% of these revisits required hospital admission. Traditionally, only attendings were asked to complete questionnaires about revisit cases.
Enrolment protocol
Patients revisiting the ED within 72 hours of initial presentation were identified via automated daily reports. Cases were considered for inclusion if an attending-trainee pair treated the patient at the index visit. When transfer of care occurred during the ED visit, the last pair prior to disposition was assigned the case. Cases were excluded if the primary author (MPG) deemed the revisit had an unrelated presenting condition from the index visit (eg, index visit was for fever, revisit was for trauma). Additionally, cases were excluded from our paired analysis if an attending cared for a patient without a trainee. The attending-trainee pairs were solicited via email within 1 day of the return visit to independently complete questionnaires. The questionnaire was designed, collected and securely managed using the REDCap electronic data capture tool. Automated email reminders and in person notification were used if questionnaires remained incomplete. By reviewing the index case, both the attending and trainee were aware of the revisit disposition before completing their independent questionnaires.
Data collection (instrument)
Our questionnaire expanded on our established case review system, whereby attendings receive notification and complete a secure, standardised questionnaire within a week of a 72-hour revisits that required hospital admission.18 We broadened the scope to include all revisits; this allowed for separate analysis of both revisit cases that are again discharged (referred to as ‘DC’ cases) and revisit cases requiring admission (‘admit’ cases). The questionnaire was refined iteratively among 12 PEM attendings and 20 trainees to ensure construct validity. Additional attendings and trainees piloted the questionnaire prior to initiating data collection.
After case specifics and participant demographics were collected, the questionnaire asked participants to indicate a primary root cause for the revisit. To standardise responses, we provided the following root cause options with examples (table 1).
Revisit root cause options with examples
Participants had the option to indicate if there was an additional or secondary root cause contributing to the revisit. Lastly, questions relevant to medical education, attitudes towards reviewing revisit cases and the time required to complete the review were solicited (online supplementary appendix 1).
Supplementary file 1
When participants selected ‘other’, investigators reviewed the response, blinded to additional case data. All three physician investigators (MPG, CEP, RGB) had to agree in order to recategorise the response as ‘potential medical deficiency’.
To safeguard protected health information, questionnaires were created and administered from within the hospital’s internal network. As this study expanded an ongoing QA process, the institutional Quality Improvement (QI) Council formally approved the project as an IRB-exempt, QI effort.
Outcome measures
The primary outcome measure was the revisit’s root cause as determined independently by the attending and trainee pair. Secondary outcome measures were respondents’ attitude towards learning and perceived value of participation in the revisit system. The balancing measure was time spent reviewing the case.
Data analysis
Descriptive statistics were used to characterise the patient, trainee and attending demographics. Frequencies of root causes selected by attendings and trainees were calculated and stratified based on revisit disposition.
In our primary analysis, we dichotomised root causes into ‘potential medical deficiency’ or ‘not potential medical deficiency’ and performed a paired analysis between attendings and trainees using McNemar’s test. We performed subgroup analyses on admitted and discharged cases. For all analyses, the primary root cause and optional secondary root cause were combined to determine any instance of potential medical deficiency. Using the attending responses as the reference standard to identify a potential medical deficiency, we calculated trainee ‘false-negative’ and ‘false-positive’ rates.
To help guide future QA and educational efforts, we performed a multivariable logistic regression to determine case-related factors that could predict attending-trainee discordance. The dependent variable was discordance and the independent variables were trainee type (paediatric resident, EM resident, PEM fellow), trainee experience (postgraduate year) and revisit disposition (DC vs admit).
Sample size
To calculate sample size, we made the assumption that the subset of ‘admit’ patients would have higher rates of potential medical deficiency and more discordance in root cause assignment as these cases could have increased acuity, complexity or the persistence of unclear symptoms pushing the revisit providers to admit. A McNemar’s sample size calculator generated 85 revisit requiring admission cases to detect a 25% rate of discordance with 80% power and an alpha of 0.05. Based on the previous year’s administrative data, approximately 30% of revisit cases were admitted. Anticipating a 60% survey completion rate, we were prepared to enrol 470 total revisit cases but planned to stop enrolling once we met our goal of 85 revisits requiring admission.
Enrolment bias analysis
Enrolment bias was assessed by comparing eligible paired cases where only the attending responded to the group of enrolled cases with both attending and trainee responses.
All statistical analyses were conducted using the software package STATA, V.13.1 (College Station, Texas, USA).
Results
Enrolment
From August 2015 to February 2016, 31 630 patients visited our ED with 1.8% (559) returning for a 72-hour revisits. Three hundred five revisit cases met inclusion criteria with 189 (62%) cases discharged (DC) and 116 (38%) hospitalised (admit) at the return visit (figure 1). Most of the exclusions were due to the fact that an attending saw the patient without a trainee. Overall questionnaire completion rate was 97% (296/305) for attendings and 73% (224/305) for trainees. Attendings and trainees completed questionnaires on the same patient—72% (218/305) for all revisits and 79% (92/116) for admits. This allowed for 218 paired cases to be included in our primary analysis.
Enrolment flow chart of attendings and trainees for paired analysis during the study period.
Study participants
The paired revisits were treated by 53 different PEM attendings; 33% had <5 years, 10% 5–10 years, 36% 11–15 years and 20% had >15 years of experience. One hundred nineteen different trainees had a patient revisit: 39% were paediatrics residents, 28% emergency medicine residents and 32% PEM fellows.
Visit characteristics
The most common chief complaint resulting in revisit was fever, followed by respiratory complaints and abdominal pain (25%, 16% and 11%, respectively). Of the paired revisits, 58% (126/218) were DC cases and 42% (92/218) were admit cases. Five cases were admitted to the intensive care unit (ICU) and four cases to the operating room.
Primary outcome
The frequency of root causes selected by attendings and trainees stratified by revisit disposition are depicted in table 2. One hundred primary or secondary root cause responses were initially free texted as ‘other’ and underwent the recoding process as outlined in the ‘Methods’ section, with 19 reclassified to ‘potential medical deficiency’. Attendings assigned potential medical deficiency to 13% of all revisits, and 16% of admits; trainees assigned potential medical deficiency to 9% of all revisits, and 13% of admits.
Proportion of return visit root causes by attendings and trainees stratified by revisit disposition. (95% CIs in parentheses)
The paired analysis of all enrolled cases is depicted in table 3a. The discordance between attendings and trainees in this analysis was 17% (38/218, 95% CI 12% to 22%). There was no significant difference in the frequency of medical deficiency between attendings (13%) and trainees (9%) (P=0.14; absolute difference 4%; 95% CI −2% to +9%). There were 29 cases where the attending selected ‘potential medical deficiency’. Using the attending response as the reference standard, the trainee false-negative and false-positive rates were 83% (24/29) and 7% (14/189), respectively.
a–c Comparison of attendings and trainees for the assessment of ‘potential medical deficiency’ as the root cause of a 72-hour ED revisits
For patients admitted at the revisit, 25% discordance was observed (23/92, 95% CI 16% to 34%) (table 3b). There was no significant difference between attendings and trainees in the dichotomised medical deficiency rate (P=0.68, absolute difference 3%; 95% CI −7% to +13%). For this subgroup, the trainee ‘false-negative’ rate was 87% (13/15).
For the subgroup of discharged cases, 12% discordance was observed (15/126, 95% CI 6% to 18%), with no significant difference between attendings and trainees in the medical deficiency rate (P=0.12, absolute difference 5%; 95% CI −2% to +12%) (table 3c). For this subgroup, the trainee ‘false-negative’ rate was 57% (4/7).
Multivariable logistic regression analysis to predict the 38 episodes of attending-trainee root cause discordance revealed increased odds of discordance in patients who are admitted at the return visit, OR 2.6 95% CI (1.3 to 5.5), while the other variables (trainee home programme and trainee postgraduate year) did not significantly influence discordance (online supplementary file 2).
Supplementary file 2
Assessment of enrolment bias
Eligible cases where attendings completed questionnaires and trainees did not (unpaired) were compared with instances where trainees and attendings both completed questionnaires (paired) about the revisit (78 vs 218 cases). The proportion of DC cases among the unpaired questionnaires did not statistically differ from that of the paired sample ((69% (54/78) vs 58% (126/218); P=0.08)). Unpaired attendings assigned 5% (n=4 cases) potential medical deficiency while the attendings in the paired sample assigned medical deficiency 13% (n=28 cases), absolute difference 8%, 95% CI (−0.5% to 14%).
Secondary outcomes
Learning
The three most commonly selected personal learning themes by attendings and trainees are included in table 4.
Three most common learning themes reported by attendings and trainees
Perceived value
Attendings and trainees rated this review process as highly valuable for both admit (attending 89%, trainee 96%) and DC (attending 86%, trainee 93%) cases. Furthermore, 57% of trainees stated interest in further education in conducting formal root cause analysis of revisit cases.
Next steps and continued improvement
Eighty per cent of attendings and 82% of trainees were highly interested in the possibility of reviewing their colleagues’ responses to the questionnaire. Attendings and trainees agreed that for most cases, the best modality to review each other’s responses would be electronically, but a notable percentage of both attendings (41%) and trainees (47%) also selected ‘in-person conversation’ if the revisit led to hospital or ICU admission.
Balancing measure: time investment
Ninety-two per cent of attendings and 91% of trainees reported completing this case review process in <10 min.
Discussion
We report the results of a 72-hour revisits QA process that incorporates the trainee perspective. This work builds on prior literature as it incorporates Croskerry’s conceptual feedback framework2 and follows best practice recommendations by Shy et al supporting trainee integration.21
We identified a meaningful rate of discordance between attendings and trainees with respect to whether or not a potential medical deficiency contributed to a patient’s revisit. Although the frequency of selecting ‘potential medical deficiency’ was similar among attendings and trainees, there were few cases in which both agreed that a potential medical deficiency was the root cause of the revisit.
Consistent with our assumptions, cases admitted at the revisit had a greater likelihood of a potential medical deficiency in care and predicted discordance between attendings and trainees in assigning potential medical deficiency. These data support the notion that revisit QA systems should focus on admit cases.21
Contrary to our hypothesis, trainees were less likely than attendings to attribute return visits to a potential medical deficiency. This finding, in addition to high trainee false-negative rates, suggests trainees would benefit from reconnecting with their attending to ensure important aspects of medical management and opportunities to improve care are discussed. Furthermore, a noticeable difference between attending and trainee responses was seen when comparing the most common learning themes uncovered through our system. We found that the attending list demonstrated a more internal locus of control, whereas the trainee responses speak to the involvement of external components of the care team.
Our rates of potential medical deficiency contributing to a 72-hour ED revisits are consistent with prior publications.14–19 The focus of prior literature addresses the value of using revisits attributed to medical deficiencies as a marker of ED quality.16 18 20 Easter and Bachur18 and Shy et al 19 transitioned this conversation from searching for the meaning of this potential medical deficiency rate to the more important discussion of how to improve care by learning from these cases. Our rates of potential medical deficiency, discordance and trainee false negatives support the review of revisit cases for advancing educational agendas, making inclusion of trainees paramount.
Prior reports support trainee involvement in QA systems and highlight learning that emerges. Platt and Coleman share their unique ‘administrative rotation’ where rotating residents review ED revisits and present findings at Morbidity and Mortality Conference. They conclude this effort is a logical way to integrate trainees into hospital QA initiatives and meet trainee requirements for patient follow-up.12 Outside of the ED setting, integration of trainees into case review processes have shown the ability to decrease hospital length of stay and costs.3 6
To our knowledge, we are the first to report on formally engaging trainees about their own 72-hour revisits cases. In doing so, we probed trainees to gauge their receptiveness to such case-based feedback. Almost all trainees considered our case review process a valuable experience and more than half expressed interest in further instruction in root cause analysis. While we believe our findings speak most significantly to the importance of trainee inclusion, we found that attending physicians also responded positively to receiving such feedback.
From this work, we strived to elicit new educational models to employ through future feedback systems. Thus, we were encouraged that both attendings and trainees shared a mutual desire to reconnect with their paired colleague around 72-hour revisits cases outside of traditional means (eg, ‘word of mouth’). This implies academic physicians, regardless of level of experience, find value in and see the learning potential from reuniting with their colleagues around revisit cases. To be most efficient, both groups report a preference for electronic technologies, although it is important to consider the role of in-person conversation for select cases.
Lastly, neither attending nor trainee groups reported excessive time burden from involvement in our system, a key balancing measure in any QI endeavour.
Limitations
There are several limitations to our work. First, it should be noted that designing a QA system around 72-hour ED revisits is arbitrary. We chose this population because of the technological and cultural feasibility to expand our established system.18
Second, our outcome measure, ‘potential medical deficiency’, is a relatively rare event but the discordance of attendings and trainees for these cases underscores the importance of joint review. Furthermore, the use of the attending’s root cause as the reference standard for calculating trainee ‘false-negative’ and ‘false-positive’ rates was not meant to imply that the attending’s opinion was the gold standard, but rather to provide another frame of analysis to understand the discordance. It should also be explicitly stated that the ultimate goal of this work was not to uncover the ‘true’ root cause of the revisit, but rather to point out that significant discordance may have both QA and educational implications. As such, use of a third party to further clarify the root cause of the revisit, while may have been informative, was never the intention of this work.
Third, while not statistically significant, we were not powered to draw conclusions from our enrolment bias analysis. Fourth, we are unable to account for how knowledge of revisit disposition may have influenced participants’ selection of the revisit’s root cause.
Fifth, the generalisability of our findings is limited by enrolling a convenience sample at a single institution as well as by the potential for recall bias. However, our active enrolment process and relatively high questionnaire completion rate supports the internal validity of our findings. Furthermore, we limited recall bias by alerting clinicians of their revisit within 24 hours.
Finally, we are unable to report on the important consideration of the trainee’s emotional reactions to participation in our system as such a question would be fraught with bias.
Conclusions
We established a QA process for 72-hour revisits that involves independent case review by both attendings and trainees. Through uncovering meaningful rates of discordance in the root cause of a revisit and by showing high trainee ‘false-negative’ rates for cases of potential medical deficiency, we support inclusion of the trainee’s perspective and focusing on revisits requiring admission to improve 72-hour revisits processes. Furthermore, our findings point the academic clinician to a novel learning opportunity across the educational continuum.
References
Footnotes
Contributors MPG is the principal investigator on this project. He was responsible for project conception, design, enrolment, analysis and writing the manuscript. He assumes complete responsibility for the submitted manuscript. MM contributed his statistical expertise around the methods and analysis plans. He assisted MPG with data management and statistical processing. He also provided critical review of the manuscript. CP was involved in project’s conception, survey drafting, recategorisation process and reviewed and edited the manuscript. RGB is the senior investigator for this project. He supervised all steps of the study and provided critical review of the manuscript. All authors approved the final version of manuscript prior to submission.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.