Article Text

other Versions

The feasibility of an interactive voice response system (IVRS) for monitoring patient safety after discharge from the ED
  1. Lisa A Calder1,2,
  2. A Adam Cwinn1,
  3. Mathieu Gatien1,
  4. Andrew Gee1,
  5. Natasha Larocque2,
  6. Samantha Calder-Sprackman2,
  7. Ria De Gorter2,
  8. Jennifer Zlepnig2,
  9. Ian G Stiell1,2,
  10. Alan J Forster2,3
  1. 1Department of Emergency Medicine, University of Ottawa, Ottawa, Ontario, Canada
  2. 2Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
  3. 3Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada
  1. Correspondence to Dr Lisa A Calder, Department of Emergency Medicine, University of Ottawa, Ottawa, ON K1N 6N5, Canada; lcalder{at}


Background Return ED visits are frequent and may be due to adverse events: adverse outcomes related to healthcare received. An interactive voice response system (IVRS) is a technology that translates human telephone input into digital data. Use of IVRS has been explored in many healthcare settings but to a limited extent in the ED. We determined the feasibility of using an IVRS to assess for adverse events after ED discharge.

Methods This before and after study assessed detection of adverse events among consecutive high-acuity patients discharged from a tertiary care ED pre-IVRS and post-IVRS over two 2-week periods. The IVRS asked if the patient was having a health problem and if they wanted to speak to a nurse. Patients responding yes received a telephone interview. We searched health records for deaths, admissions to hospital and return ED visits. Three trained emergency physicians independently determined adverse event occurrence. We analysed the data using descriptive statistics.

Results Of 968 patients studied, patients’ age, sex, acuity and presenting complaint were comparable pre-IVRS and post-IVRS. Postimplementation, 393 (81.7%) of 481 patients had successful IVRS contact. Of these, 89 (22.6%) wanted to speak to a nurse. A total of 37 adverse events were detected over the two periods: 10 patients with 10 (6.5%) adverse events pre-IVRS and 16 patients with 27 (16.9%) adverse events post-IVRS. In the postimplementation period, the adverse events of seven patients were detected by the IVRS and five patients spontaneously requested assistance navigating post-ED care.

Conclusions This was a successful proof-of-concept study for applying IVRS technology to assess patient safety issues for discharged high-acuity ED patients.

  • emergency department
  • safety

Statistics from

Key messages

What is already known on this subject?

Interactive voice response systems (IVRS) have been used in many areas of healthcare from population health screening to medication management. To date, the use of this technology among ED patients has been very limited. Currently, it is estimated that 7%–8% of ED patients experience adverse events after discharge; yet, gaps in the continuity of care post-ED visit have been highlighted as a patient safety concern.

What this study adds

In this before and after study of high-acuity patients discharged from an ED in Canada, the use of an IVRS was able to reach 81.7% of patients, of whom 22.6% desired an interview with a nurse. Nearly half of adverse events were detected via the IVRS and five additional patients needed help navigating care. The IVRS is therefore a feasible method for improving ED care.



What is the problem?

High-acuity patients who are discharged from EDs are potentially vulnerable to patient safety events. In Canadian EDs, we discharge 84% of our patients who are triaged as having a high-acuity problem.1 There are two key gaps in post-ED discharge care: (1) patients may be uncertain of what to do when symptoms recur, persist or worsen and (2) follow-up consultations that are arranged may not occur due to a system failure or over-reliance on patients’ memory. In addition, while many patients do well postdischarge, an important proportion, estimated to be 7%–8%, suffer adverse events after discharge.2 3 Adverse events are adverse outcomes related to healthcare provided rather than progression of disease.4

Traditionally, adverse events have been studied via incident reporting, morbidity and mortality rounds or chart review.5 Prospective surveillance has been proposed as a superior method to detecting the occurrence of adverse events. This can be achieved by electronic triggers for patients in hospital or telephone follow-up for those who have been discharged.5–7

Increasingly, telephone follow-up for discharged patients has been promoted as a method to monitor patient safety as well as patient satisfaction.8–10 Telephone follow-up offers an opportunity to close the gaps that occur in post-ED discharge care. It may further allow the identification of patients who experience adverse outcomes such as unscheduled admission to hospital or ED visits. One of the fundamental challenges to implementing telephone follow-up is the resource-intensive nature of contacting every patient discharged from busy EDs.

Interactive voice response systems (IVRS) present an opportunity to perform telephone follow-up in a more focused and less resource intensive way. IVRS are technologies that connect people to computers using a telephone. They consist of hardware and software that translate human telephone input into digital data. Users are prompted by way of prescripted dialogues to provide information by touching the keypad of the telephone or through voice recognition. IVRS have recently been applied in many healthcare settings including after hospital discharge for encouraging adherence to tests, treatments and behaviours for patients with asthma, patients with heart failure, patients on anticoagulants and smokers with coronary disease.11–15 Thus far, there have been limited applications in the ED, the only published data we found were researchers who used IVRS as a call-in service to allow for confidential data collection for patients who have experienced assault or partner violence.16 17 It is thus uncertain how feasible it is to use IVRS for telephone follow-up and adverse event detection. Furthermore, there is no consistently used method to detect adverse events after ED discharge.

Study objectives

We undertook this study to explore the feasibility of IVRS for following high-acuity ED patients after discharge. Our overall aim was to determine whether IVRS facilitated telephone follow-up of patients discharged from the ED. Specifically, our objectives were to (1) pilot IVRS and evaluate for feasibility; (2) determine the proportion of discharged emergency patients who experienced adverse events within 14 days of the index ED visit before and after the introduction of IVRS; (3) determine the preventability, severity and type (diagnostic issue, management issue, disposition issue, suboptimal followup) of adverse events.


Study design and setting

This before and after interventional pilot feasibility study was conducted at the Civic Campus of The Ottawa Hospital, a tertiary care, academic hospital with an annual ED census of 75 000 patient visits per year, 83.7% of these are triaged as Canadian Triage and Acuity Scale (CTAS) 1–3 on a five-point scale.18 The Ottawa Hospital has two EDs at two of its three campuses, and the electronic health records system is shared across all three campuses. First, we prospectively assessed for the occurrence of adverse events during a 2-week period prior to the introduction of IVRS in July 2012. Then, we introduced the IVRS and telephone follow-up system and followed patients for 2 weeks postintervention. This study was approved by The Ottawa Hospital Research Ethics Board. All data collection tools and analysed records were de-identified to preserve the confidentiality of personal health information.

Study population

The study population of interest was consecutively discharged ED patients from high-acuity areas of the ED before and after the implementation of the IVRS system. At The Ottawa Hospital, high-acuity areas are geographical areas of the ED where patients who require stretchers are placed. These include monitored and unmonitored beds. We included all types of emergency clinical presentations. We excluded any patients who were admitted on the index ED visit, did not have a touchtone telephone or telephone contact, lived in a nursing home, had cognitive impairment and no substitute decision maker available, did not speak English or opted out of the study on telephone contact. While it is possible these excluded patients were at risk for adverse events, we were unable to include them due to the nature of the intervention.

Patient screening

We asked registration clerks to attach an information card to the chart for all patients triaged to the high-acuity areas of the ED (see figure 1). The Research Ethics Board waived the requirement of informed consent for initial enrolment in this study and patients were not directly approached during their ED visit. The research assistant screened all ED visits daily for eligible patients using the electronic health records database. On identifying eligible patients, the research assistant collected baseline data from the ED record.

Figure 1

Methods flow. IVRS, interactive voice response system.


The IVRS service (Telask Technologies) was programmed to start calling discharged ED patients on the second postdischarge day. The caller identification was set to display ‘Ottawa Hospital’. The IVRS system first determined if the person who answered the telephone was the patient in question. If the person who answered the telephone was not the patient, the IVRS system asked to speak to the patient. If the patient was not available, the system would advise the person to hang up. No reason for the call was disclosed. If the call was answered by the wrong person, or answered by an answering machine, the IVRS recorded this information and called back repeatedly (maximum of seven times) between 9:00 and 21:00 for 3 days. Once a call was answered, the correct identity of the patient was verified by asking the patient to press ‘1’ for yes and ‘3’ for no if they identified as the name of the patient.

Once the patient’s identity was confirmed, a two-question survey was administered: (1) Are you having a health problem? (2) Do you want to speak to a nurse? The survey questions were recorded by voice talent provided by Telask Technologies. Patients were asked to press ‘1’ for yes and ‘3’ for no. At the end of the call, regardless of a yes or no response, a recorded message indicated that if the patient had ongoing health concerns, they should contact their family physician or Telehealth Ontario, a provincial telephone service where the public can access over the phone advice for health concerns.

If a patient answered ‘yes’ for wanting to speak to a nurse, the IVRS sent an email to an ED nurse research assistant who attempted to contact the patient the next business day and perform a structured telephone interview (see online supplementary appendix A). The nurse asked about ED visits or admissions within the last 2 days, and whether the patient had visited a healthcare provider or experienced any new or worsening symptoms. If a patient identified worsened or new symptoms and had not seen a healthcare provider, the nurse reminded the patient of the option of seeing a primary healthcare provider or returning to the ED. She did not explicitly ask if patients were having difficulty navigating the healthcare system, but ended the interview by asking: "Do you have any comments for us?”. If the nurse was unable to reach the patient, she would attempt to contact them a further six times. If unable to reach the patient after six attempts, there were no further call attempts.

Supplementary file 1


At 14 days postdischarge, the research assistant verified via the electronic health record whether the patient experienced a flagged outcome. A flagged outcome was defined as any of the following: deaths, admissions to hospital, return ED visits, or visits to a healthcare provider. Patients enrolled in the post-IVRS phase who did not request an interview and had no record of returning to hospital received a second telephone follow-up call from a nurse research assistant at 14 days to determine the occurrence of a flagged outcome.

For both phases, we searched the electronic health records database to identify flagged outcomes for all enrolled patients, irrespective of whether they were contacted by the research nurse in the second phase. We employed registered nurse research assistants to compile each flagged outcome into a case summary.

From the flagged outcome case summaries, we determined those that were adverse events. Adverse events were any death, unscheduled admission, ED visit or unscheduled healthcare provider visit which was deemed associated with healthcare provided rather than progression of disease.4 An example of an adverse event would be a patient discharged from the ED with a missed myocardial infarction (see figure 2 for definitions). All flagged outcome case summaries were independently reviewed by three trained emergency physicians (LC, AG, MG) blinded to patient and treating physician identities. We trained these event reviewers using a piloted programme consisting of adverse event definitions review and case examples, detailed explanation of the adverse event review process, and feedback on first 10 then next 10 adverse event reviews. We used an adapted adverse event determination method from the Harvard Medical Practices study with which we have extensive experience from previous adverse event studies.2 3 11 19–21

The adverse event determination process consisted of structured questions to guide the reviewer to assess if the flagged outcome was related to healthcare received rather than progression of disease (see online supplementary appendix B for questions). Each reviewer (LC, MG, AG) rated their level of confidence that the flagged outcome was related to ED care using a six-point Likert scale (0=no evidence for causation, 6=certain evidence for management causation). If two out of three reviewers had a level of certainty greater than 4/6 (ie, 5/6 or 6/6), we classified the flagged outcome as an adverse event. Disagreements were resolved by consensus. We used one reviewer (LC) to determine preventability and severity for all adverse events. An adverse event was deemed preventable if there was a clearly modifiable factor in care such as diagnostic error, ED management error or medication adverse effect (see figure 2 for definitions). For example, accurate detection of pneumonia on x-ray would likely have ameliorated the outcome.


Our outcomes for the primary objective were feasibility outcomes, including ratio of successful calls to all calls made by IVRS and proportion of patients successfully completing the IVRS survey. The outcome for the secondary objective was the occurrence of adverse events before and after the IVRS intervention, and the outcomes for the tertiary objective were adverse event type and severity.


We report the proportion of adverse events and preventable adverse events with 95% CIs. We describe the study population using descriptive statistics: mean and SD for continuous variables with normal distribution or median and IQR for skewed distributions; frequency and proportion for categorical variables. Differences in the proportion of adverse events before and after IVRS were measured using a Χ2 test.

Sample size

We calculated our sample size for this feasibility study to be 1000 patients. With an estimated adverse event proportion of 5% based on previous research, we anticipated that 50 adverse events would occur with a 95% CI of 1.4% on either side of the point estimate for the proportion of adverse events.2


We enrolled a total of 968 patients: 487 patients in the pre-IVRS and 481 post-IVRS implementation. Thirty two (3.2%) of the 1000 eligible patients were excluded due to admission on the index visit, opting out, telephone numbers that were either disconnected or restricted (figure 3).

Figure 3

Data flow.  IVRS, interactive voice response system. 

The baseline characteristics of both cohorts were similar (table 1). The overall mean age was 57 and just over half were female (n=505, 52.2%). The cohorts were mostly CTAS 1–3 (n=924, 95.4%) and presented with common chief complaints such as chest pain, abdominal pain, palpitations and shortness of breath.

Table 1

Baseline characteristics for 968 enrolled pre-IVRS and post-IVRS patients

Feasibility outcomes

We were able to pretest and implement the system within 3 weeks. We did not experience any technical issues during the 2-week period that the IVRS system was operating. After IVRS implementation, we were able to successfully contact 393 (81.7%) of patient with the IVRS system (table 2). Of these, 107 (27.2%) wanted contact with a registered nurse. Our nurse was successful in contacting 89 (83.2%) of these patients of whom 76 (85.4%) wished to proceed with the interview (requiring a total of 16.5 hours to complete). Of note, this meant that 286 (72.8%) patients did not desire an interview.

Table 2

Feasibility data for IVRS implementation among 481 eligible patients

Secondary outcomes

Based on our electronic health records review, we detected 155 (31.8%) flagged outcomes in the pre-IVRS phase and 160 (33.3%) in the post-IVRS phase. During our adverse event analysis, we detected a total of 37 adverse events (3.8%, 95% CI: 2.6% to 5.0%): 10 adverse events (10/155=6.5%, 95% CI: 3.1% to 11.5%) pre-IVRS and 27 adverse events (27/160=16.9%, 95% CI: 11.4% to 23.6%) post-IVRS. These adverse events were experienced by 10 and 16 patients, respectively, a non-significant difference 2.1% (95%CI: 1.0% to 3.7%) vs 3.3% (95%CI: 1.9% to 5.3%, p=0.2). The proportion of preventable adverse events was similar for both groups: 9 (90.0%) and 24 (89.0%), respectively (table 3). All adverse events in the postimplementation group were detected by health records review. However, of the 16 patients with adverse events, 7 patients experiencing adverse events requested a phone interview, while 9 patients who did not receive a telephone interview experienced adverse events. None of these 9 patients had requested a telephone interview by the IVRS system, nor did they have an unlisted phone number, but all received an IVRS contact.

Table 3

Characteristics of 37 adverse events identified pre-IVRS and post-IVRS

Diagnostic issues and unsafe disposition decisions were the most common adverse event types (table 3). Most adverse events (26, 70.3%) resulted in return ED visits or admissions to hospital. There was one patient in the postphase who died, an elderly patient who lived alone and presented to the ED with a recent history of increasing falls. This patient was assessed for injuries; none were detected and discharged home. The patient returned to the ED 8 days later after a fall resulting in a hip fracture, developed postoperative pneumonia and delirium and died in hospital. This was classified as unsafe disposition decision making and suboptimal follow-up (see online supplementary appendix C for further examples of adverse events). Thirty three of the 37 adverse events were considered preventable. The system issue of inadequate monitoring of patient’s treatment postdischarge was relevant in 13 cases.

Our nurse interviewer did not refer any patients to the ED during her telephone follow-up calls. She did refer a small proportion (n=4) to see their family physicians for minor issues. She also noted 5 of the 74 patients spontaneously described challenges navigating the healthcare system and helped these patients with their follow-up arrangements. Four patients also spontaneously indicated dissatisfaction with their care. And 10 patients spontaneously expressed a high degree of satisfaction with their care.


We found IVRS to be feasible, easy to set up and run for our ED population. We successfully contacted most patients while saving the resource intensity required for consecutive manual telephone follow-up. We detected almost half of the adverse events identified with health records review. The latter approach is a cost-prohibitive method for continuous prospective adverse event surveillance for most Canadian EDs; thus, IVRS is an appealing alternative. As with previous ED adverse event studies, we found a high degree of preventability at >80%.2 20 21 In addition to detecting adverse events, we also uncovered the system issue of gaps in continuity of care. Our nurse intervened with 9 of the 89 patients contacted by referring them to their family physician or helping secure follow-up arrangements. Without the IVRS system, these patients may have been lost to follow-up after their index ED visit since our ED does not routinely engage in health records review adverse event surveillance.


Prior ED studies have used traditional telephone follow-up (nurses manually phoning patients) and shown an improvement in the likelihood of the follow-up of elderly patients with family physicians after ED discharge and a decrease in adverse events in high-risk ED patients.6 7 IVRS has been shown to be feasible for urban populations and has been used to collect data on ED populations in whom confidentiality was important (eg, patients at risk for ethanol abuse and partner violence).16 17 22 Outside the ED, patients recently discharged from the hospital found that IVRS implemented for postdischarge follow-up was easy to use and they did not necessarily prefer a call from a live person.3 To our knowledge, our study is the first to report on the feasibility of using IVRS and its ability to detect adverse events and identify continuity of care gaps after ED discharge. We do not know if we prevented any adverse events and our system was not explicitly designed for this purpose. Future work could examine how to develop this more fully. It would appear that there is certainly opportunity to develop further applications of IVRS for ED populations. Particular attention to the use of this system to strengthen the security of follow-up arrangements could be beneficial.


This was a single centre pilot study with a small number of adverse events. Just under twenty per cent of patients were not reached and while this is consistent with previously reported ED manual telephone follow-up response rates (68%–82%), this could be improved by asking ED patients to identify their best contact number and time of day. Given only seven patients with adverse events requested interviews, this provokes the question of whether patients can appropriately self-select for a call from a nurse based on their clinical condition. This deserves further exploration, perhaps by improving the IVRS scripting and prompts. While a randomised approach could reduce selection bias, we did not randomise patients to the IVRS because of the increased cost of administering the IVRS randomly rather than for consecutive patients during a specified time period. We relied on health record documentation for adverse event determination which may have resulted in an underestimate of adverse events. Since post-IVRS patients received an additional telephone call to ensure accurate adverse event determination, it is possible we may have detected more adverse events this way. That being said, using IVRS we identified only 43.8% of patients with adverse events as detected by health records review, suggesting that detailed health records review may yield a greater proportion of adverse events. It is possible some patients returned to centres other than our own and that this was not detected by our study, although our previous studies have indicated that this rarely occurs2 21

Research implications

We confirmed that IVRS can be successfully used in high-acuity ED patient populations and we believe that this technology holds promise for monitoring continuity of care after discharge. We also believe that this method is less resource intensive than consecutive manual telephone follow-up or detailed health records review. Future work should include a randomised controlled trial to evaluate the impact on safety outcomes and consider other interventions such as specifically asking about follow-up arrangements as part of an IVRS bundle. There is also opportunity to combine approaches examining quality of care with assessing patient satisfaction using IVRS.


We conclude that IVRS can be used to monitor patients post-ED discharge for patient safety outcomes and that this technology is feasible for use in this setting. Using technology to screen patients for follow-up was efficient, meaning that this is a powerful tool for use in quality improvement efforts in the ED.


We thank Peter Fallis, Vincent Talbot and Kingsley Yu fromTelask for providing, setting up and administering the IVRS. Sincere thanksalso to Marlene Beehler our Research Nurse Assistant and Angela Marcantonio, ResearchEthics Coordinator. We also thank Cathryn Peloso, Research Assistant for studyset up and Teena Ostapyk, ED registration clerk manager, Anna Zielinski and allCivic ED clerks for providing patients with study information forms.


View Abstract


  • Canadian Association of Emergency Physicians Annual Conference 2014

  • Contributors Each author has contributed substantially to the production of the manuscript as follows: LAC is responsible for idea for the project, methods, data analysis, revising manuscript and integrity of paper as a whole. AAC involved in data analysis and revising the manuscript. MG involved in data collection and revising the manuscript. AG involved in data collection and revising the manuscript. NL involved in data collection, data analysis and revising the manuscript. SC-S involved in data collection and revising the manuscript. RDG involved in data collection and revising the manuscript. JZ involved in data collection and revising the manuscript. IGS involved in data analysis and revising the manuscript. AJF involved in data analysis and revising the manuscript.

  • Funding Canadian Medical Protective Association.

  • Competing interests None declared.

  • Ethics approval Ottawa Hospital Research Ethics Board.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.