Article Text

Download PDFPDF

Development of the Rapid Assessment, Prioritisation and Referral Tool (RAPaRT) for multidisciplinary teams in emergency care settings
  1. Steven M McPhail1,2,
  2. Angela Vivanti3,4,
  3. Kate Robinson5
  1. 1Centre for Functioning and Health Research, Metro South Health, Brisbane, Queensland, Australia
  2. 2Institute of Health and Biomedical Innovation and School of Public Health and Social Work, Queensland University of Technology, Brisbane, Queensland, Australia
  3. 3Nutrition and Dietetics Department, Princess Alexandra Hospital, Brisbane, Queensland, Australia
  4. 4School of Human Movement Studies, The University of Queensland, Brisbane, Queensland, Australia
  5. 5Speech Pathology Department, Princess Alexandra Hospital, Brisbane, Queensland, Australia
  1. Correspondence to Dr S M McPhail, Centre for Functioning and Health Research, PO Box 6053, Buranda, QLD 4102, Australia; steven.mcphail{at}health.qld.gov.au

Abstract

Objective This investigation utilised the expertise of allied members of multidisciplinary teams working in emergency care settings to develop and validate a Rapid Assessment Prioritisation and Referral Tool (RAPaRT). This instrument is intended for use among patients (with non-life threatening acuity) presenting to emergency care settings to indicate when referral to an allied member of the multidisciplinary team is warranted.

Method This three stage instrument development and validation study included: a Delphi panel process to determine key criteria to guide instrument development and identify potential items to be carried forward for testing (stage 1); a prospective cohort of consecutive admissions (n=153) to investigate item sensitivity and specificity and retain only the most suitable items (stage 2); then final consultation with the Delphi panel to ensure the final instrument was clinically amenable (stage 3).

Results 23 potential items were identified following stage 1. At the completion of item sensitivity and specificity analysis and in consultation with the Delphi panel, seven items were retained in the instrument. Area under the receiver operating characteristic curve was 0.803 for these seven items in predicting when a referral was warranted. Final consultation with the Delphi panel members also resulted in the addition of an open ended (eighth) item to allow description of any infrequent, but important, reason for referral.

Conclusions The RAPaRT has demonstrated substantial promise as an efficient clinically amenable instrument to assist multidisciplinary teams in emergency care settings. Further research to investigate the wider implementation of the RAPaRT is warranted.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Key messages

  • What is known on this subject

  • Multidisciplinary teams have been expanded to include more allied (non-medical or nursing) clinical staff in emergency care settings. Their role in this setting may be useful in the management of patients with high levels of complexity due to chronic health conditions or social circumstances. However, an instrument to facilitate decisions for referral to allied team members has been lacking.

  • What this study adds

  • This investigation has reported the development of the Rapid Assessment, Prioritisation and Referral Tool (RAPaRT), which has demonstrated promise as an instrument to assist referral decision making within multidisciplinary teams in emergency care settings.

Background

The demand on hospital services has been consistently rising internationally.1 ,2 It has been reported that the number of emergency department presentations has been growing at approximately 3–6% per year among high income nations.3–6 In the Australian public healthcare setting, the rate of emergency department presentations has increased by 32% between 1999 and 2009, representing a 3.6% annual growth rate after adjusting for population changes.1 An undesirable consequence of increased demand may include overcrowding and delays,7–10 diverting of ambulances,7 increased risk of adverse outcomes11 and failure to meet performance targets that may be linked to government funding.

Two related strategies for managing demand and improving emergency care that have been implemented in some locations include the expansion of multidisciplinary teams in emergency settings to include more allied (non-medical or nursing) clinical staff, as well as the expansion or introduction of short stay units within or alongside emergency departments.12–14 These strategies may be particularly useful for patients who do not have a high level of acuity (their presentation is not immediately life threatening or an admission to a traditional hospital ward may not be required), but due to the complexity related to their health state or social circumstances may require multidisciplinary services to either be delivered or arranged prior to discharge from hospital. A primary objective for these services is to offer efficient and effective multidisciplinary interventions to help patients in the short (and long) term while preventing further emergency presentations and avoidable healthcare utilisation.14

To achieve the necessary efficiency for success in these complex, fast paced environments it is essential that patients are assessed expeditiously and appropriately referred to health staff, including allied members of the multidisciplinary team. Due to the emerging nature of multidisciplinary practice in this area, there is currently no validated rapid assessment instrument to assist in identifying which patients should be seen by allied members of the multidisciplinary team. The purpose of this investigation was to utilise the expertise of allied members of multidisciplinary teams working in emergency care settings at a tertiary facility to develop a Rapid Assessment Prioritisation and Referral Tool (RAPaRT V.1.0). This instrument is intended for use among patients (with non-life threatening acuity) presenting to emergency care settings to indicate when referral to an allied member of the multidisciplinary team is warranted.

Design

A three stage instrument development and validation study was implemented (figure 1). The first stage included a Delphi panel process to allow clinical experts to determine key criteria to guide instrument development and identify potential items to be carried forward for testing. The second stage included pretesting of the potential items among clinical staff and patients to ensure item wording was clear and interpreted as intended (stage 2a) and then completion of the items among a prospective cohort of consecutive admissions to investigate item level sensitivity and specificity (stage 2b). The third stage included discussion of outcomes from stage 2 with the panel members for final consensus and decision making to ensure the final structure of the instrument was clinically amenable. This investigation was approved by the Metro South Human Research Ethics Committee.

Figure 1

Three stage study design.

Delphi panel process (stage 1)

Stage 1: methods

The Delphi panel members were senior clinicians identified and invited to participate during departmental staff meetings at the tertiary hospital where the instrument development was taking place. The Delphi panel (n=12) included two representatives from each of the participating health disciplines: dietetics, occupational therapy, physiotherapy, pharmacy, speech pathology and social work. All panel members had experience working in emergency care settings. The Delphi process was facilitated by members of the research team (SMM, AV, KR).

The Delphi panel was charged with two tasks: defining overarching desirable attributes of the instrument to ensure the final instrument would be clinically amenable (task 1) and identifying potential items to be carried forward for subsequent testing (task 2). Panel members were provided with written stimulus material prior to the first Delphi panel round. This included items from the Identification of the Seniors at Risk (ISAR)15 instrument and the questionnaire of Rowland,16 which have previously demonstrated ability to predict emergency presentations.

In the first round, the Delphi panel members were asked to provide comments and suggestions for tasks 1 and 2. This included identifying additional items (or instruments) that would likely indicate a referral to their clinical discipline was warranted. The Delphi panel members were given a 2 week window to provide their responses to the panel facilitators. The panel facilitators compiled a written summary (with comments anonymised to preserve confidentiality) for distribution to the panel prior to the second round. During the second round, panel members were given the opportunity to comment on the summary of proposed desirable attributes and proposed items, as well as offer suggested modifications. This process was repeated until the panel were satisfied with the written summary of guiding principles and potential items to be carried forward after the third wave of Delphi panel feedback.

Stage 1: results

Three criteria were identified to ensure the clinical amenability of the RAPaRT instrument. First, the final instrument would need to be short and simple enough to permit completion for a high proportion of patients; ideally, a maximum of 10 items, with priority given to items that had potential for identifying referral to multiple discipline groups. Second, each item would need to be ubiquitously understood by health professionals from any discipline likely to work in this clinical setting (and their assisting or administration staff), and ideally even by patients to enable self-completion. Third, the final instrument should exclude the majority of patients who do not warrant referral (true negatives) while minimising the number of false negatives (inappropriate referrals). However, in order to maximise the number of patients who warrant referral being captured (true positives), the Delphi panel considered sensitivity to be more important than specificity when deciding which items to retain in the final instrument.

At the completion of stage 1, a total of 23 potential items were identified for further testing during stage 2. These items included all seven items from the Rowland instrument16 and all six items from the ISAR instrument.15 Additionally, the Malnutrition Screening Tool17 was proposed as a single item (with three questions) as well as nine additional single question items from the Delphi panel members. For ease of response interpretation, each item was worded to ensure either a yes or no response would be appropriate, with the exception of the Malnutrition Screening Tool (which retained the question quantifying the amount of weight loss).

Cognitive pretesting (stage 2A)

Stage 2A: methods

Pretesting of survey instruments using cognitive methods has been recommended to ensure that questions meet the purpose for which they are intended and are understood in the intended way.18 The items carried forward from stage 1 were pretested among one clinician member from each of the six participating professional disciplines, a resident medical officer, a nurse, an allied health assistant and an administration assistant, and then patients (n=5) at the participating hospital. A ‘think aloud’ approach (in an interview format) was used to explore what respondents thought each question was asking. This involved respondents verbally describing their thought process while reading the items and formulating their responses.18 This was followed by a question asking for suggestions to improve item wording.

Stage 2A: results

This pretesting indicated all 23 items were universally and correctly understood. The primary refinement to arise out of the cognitive pretesting was an alteration to the item about falls in the last 6 months—to not only report if the patient had fallen, but the total number of falls they had experienced in the past 6 months.

Prospective cohort (stage 2B)

Stage 2B: methods

The prospective cohort included consecutive weekday admissions to the medical assessment and planning unit adjoining the hospital emergency department at the participating facility. This unit accepts patients with emergency presentations classified in the lower levels of acuity but who are not appropriate for immediate discharge from the emergency department, frequently due to issues relating to social complexity, multiple chronic comorbid conditions or other difficulties coping in the community that contributed to the emergency presentation. The target length of stay in this unit is 0–72 h; hence patients must be quickly assessed and referred on for appropriate follow-up interventions (delivered either in hospital or elsewhere following discharge). These assessments and subsequent interventions frequently include allied members of the multidisciplinary team.

Given the Delphi panel considered sensitivity to be more important than specificity, sensitivity was used for the sample size calculation. The sample size calculation for an anticipated sensitivity of 0.850 for the final instrument to detect that a referral was warranted (desired precision of 0.075, assuming 95% confidence level and 0.600 prevalence estimate (based on local audit data) of referral being warranted) indicated that 147 complete datasets were required. A target sample size of 153 patients was then selected to allow for up to six incomplete datasets (due to any unforeseen medical emergencies).

Stage 2B: results

A total of 153 patients completed the questionnaire, including all 23 potential items administered by a research assistant. A clinical staff member (blind to the questionnaire responses) from each of the six participating disciplines then assessed each patient to determine whether a referral to their discipline was warranted. In addition to the questionnaire items, participants’ age, gender and presenting complaint were recorded as clinical demographic variables to describe the sample (table 1). The characteristics of this consecutive patient sample were consistent with the characteristics of patients seen in this unit over a calendar year (n=4714, including mean (SD) age 63 (18.3) years and 55% male).

Table 1

Demographic and clinical information of the prospective cohort

All patients completed the 23 potential items and were assessed by each of the six allied disciplines (100% completion). Sensitivity and specificity for each of the 23 items (in indicating whether a referral was required, as determined by the allied team members) was calculated separately for each of the six allied professional disciplines (table 2). Item reduction was driven by these sensitivity and specificity values, with priority given to items that indicated referral to more than one discipline and with a lower threshold for sensitivity than specificity (to maximise the sensitivity of the final instrument), as well as priority given to items with greater utility (indicating referral to more than one allied team member). To this end, six individual items had a sensitivity and specificity greater than 0.40 and 0.70, respectively, for two or more professions (see table 2). These items were retained in the instrument. Additionally, the investigators noted two items (taking more than three medications and seeing well) had sensitivity values greater than 0.40 for all professions (but lower specificity values) and decided to seek advice from the panel members to determine whether either (or both) of these items was worthwhile retaining.

Table 2

Item sensitivity and specificity values for indicating whether a referral to each of the six participating allied (non-medical or nursing) disciplines was warranted

Post cohort consensus consultation with panel members (stage 3)

Following completion of the consecutive cohort of admissions, a member of the research team (SMM) facilitated a group discussion with the panel members to finalise the instrument structure prior to the final analyses. Panel members were in agreement that the six items retained on the basis of sensitivity and specificity values were suitable for the final instrument. When considering the two items with sensitivity values greater than 0.40 for all disciplines (but <0.70 specificity), no discipline group considered the item dealing with ‘seeing well’ as a priority for indicating a referral was warranted to their discipline; this item was dropped from the final instrument. In contrast, the item asking about whether patients take three or more medications was considered worthwhile retaining, particularly if the instrument were to be used in an emergency setting where a blanket referral for a pharmacist consultation was not commonplace. This meant that seven items were retained from the 23 potential items.

The panel also considered a proposal to include an additional item to enable the instrument administrator to capture anything unusual that would clearly indicate a referral was warranted, but may not be captured by the seven retained items. This occurred after a member of the panel highlighted that items pertaining to infrequent, but important, referral indicators (like suspected neglect, abuse or exploitation) were unlikely to occur frequently enough to warrant retention in a short instrument as a standalone item. The panel members reached consensus that the utility of the instrument would likely be enhanced by the inclusion of an open item enabling the administrator to specifically describe any other important reason for referral.

During discussion regarding the referral pathways to be linked to each item, the panel members reached consensus that the practicality of precise referral pathways was likely to be context specific for facilities that may use the instrument. Therefore, perhaps referral pathways should be determined locally by multidisciplinary teams considering using the instrument, rather than a one size fits all approach. For example, a patient presenting following a fall may require different referral pathways dependent on the availability of physiotherapists or occupational therapists in the emergency setting, or potentially other defined referral pathways, such as a local falls clinic.

Predictive ability and referral trigger

The ability of the combined final seven items retained from the cohort to predict whether a referral to an allied member of the multidisciplinary team was warranted was examined using the area under the receiver operating characteristic curve (ROC curve). This analysis indicated that the area under the curve (AUC) for the seven retained items was 0.803 (95% CI 0.730 to 0.876) which, according to the interpretation proposed by Swets,19 could be interpreted as ‘good’. To gauge whether this was likely to be an improvement on the original stimulus items alone, this analysis was also repeated for the ISAR (AUC=0.722, 95% CI 0.645 to 0.799) and Rowland instrument items (AUC=0.706, 95%CI 0.641 to 0.770), which both performed in the ‘fair’ range for indicating a referral was warranted. This indicated that the seven items retained in the RAPaRT instrument would likely be better suited to the purpose of indicating when a referral was warranted than the ISAR or Rowland instruments among this sample.

The ROC curve for the seven retained items (figure 2) demonstrated noticeable loss of sensitivity with increasing number of positive responses required to trigger a referral. Sensitivity, specificity and total correct classification values were calculated for one, two or three positive responses as potential referral trigger cut-offs for the seven retained items (table 3). The one or two positive response referral trigger cut-offs would have both resulted in approximately 75% of patients in this sample being correctly classified, although the sensitivity was higher for a single positive response being required to trigger a referral (0.874).

Figure 2

Receiver operating characteristic curve (ROC curve) for the seven retained items, with dots numbered as cumulative total of positive responses.

Table 3

Performance of potential minimum positive response cut-offs as a referral trigger (for indicating whether referral was warranted) from the seven retained items (bold items in table 2)

Discussion

This three phase investigation concluded with the retention of seven specific items in the RAPaRT (V.1.0) instrument and an additional open ended item to describe any other important reason for referral. The seven retained items demonstrated a promising ability to differentiate between patients who did and did not warrant referral to allied members of the multidisciplinary team. Well resourced multidisciplinary teams treating patients with non-life threatening acuity in emergency care settings may elect to use the RAPaRT instrument to trigger an allied referral pathway from a single positive response. Less well resourced teams may elect to use a two positive response cut-off before a referral is triggered, which would likely eliminate some inappropriate (but also some appropriate) referrals (table 3).

Given the variability between emergency care facilities in the composition and availability of allied members of the multidisciplinary team, the specific referral pathways generated from the RAPaRT instrument may require tailoring (and evaluation) in local contexts. Potential options for referral pathways may include: specific questions automatically triggering referrals to specific professional disciplines (table 2 values may be of assistance in this regard); a positive response to any item may trigger a team leader (medical, nursing or allied health) to view the RAPaRT responses and generate referrals as they see fit; or a combination of these approaches.

Limitations

The findings from this investigation are promising but must be interpreted with caveats. First, this study was conducted at a single site. Validation of the instrument among external sites and with independent samples is a priority. Second, while consecutive patients presenting to this tertiary centre are likely to be typical of patients with non-life threatening acuity presenting at other large public hospital facilities, this multidisciplinary team included six non-medical or nursing health disciplines that may not all be represented in emergency care settings at other facilities. Third, questions were interview administered by a research assistant; patients may have provided different responses if the instrument was self-administered. Finally, the addition of the open ended item to capture any other reason for referral (not identified by the retained seven items) may further improve the sensitivity of the instrument. This may (or may not) have a detrimental impact on the specificity of the instrument but should be a focus of further research.

Future research

Investigations of alternative referral tree structures across a range of sites may prove useful. This may include validation of certain referral pathways generated from specific items, or assignment of priority values to referrals to assist in referral triage. Investigation of pragmatic aspects of RAPaRT implementation may include determining the most efficient (and reliable) method of completing the instrument, where self-administration, proxy administration, health professional administration or administration by an allied health assistant are worthy of consideration.20 The impact on health service outcomes (including patient flow, volume and timing of referrals), as well as patient outcomes,21 from implementing the RAPaRT in emergency health services, are also important future research directions. This may include evaluation of the clinical and economic impact of using a single positive response to trigger a referral versus two or more positive responses based on real life clinical data. This would help inform discussion on the acceptability (or otherwise) of using a two rather than a one positive response cut-off to trigger a referral in order to increase the specificity of the instrument, at the expense of sensitivity, among this clinical group potentially at high risk of subsequent hospital presentations.22

Conclusion

The RAPaRT has demonstrated substantial promise as an efficient clinically amenable instrument to assist multidisciplinary teams in emergency care settings. Further research to investigate the wider implementation of the RAPaRT is warranted.

Acknowledgments

The authors would like to acknowledge the valuable contributions from the clinical staff involved in each stage of instrument development.

References

Footnotes

  • Contributors SMM contributed to study conception, design, data collection, analysis, principal manuscript drafting, appraisal and editing. AV contributed to study design, data collection, manuscript drafting, appraisal and editing. KR contributed to data collection, and manuscript appraisal and editing.

  • Funding This project was supported by a health practitioner research grant from the Queensland Department of Health. SMM is supported by a National Health and Medical Research Council (of Australia) fellowship.

  • Competing interests None.

  • Ethics approval The study was approved by the Metro South Human Research Ethics Committee.

  • Provenance and peer review Not commissioned; externally peer reviewed.