Article Text

Download PDFPDF
Agreement and predictive value of the Rockwood Clinical Frailty Scale at emergency department triage
  1. William Shrier1,
  2. Colin Dewar1,
  3. Piervirgilio Parrella2,
  4. David Hunt3,
  5. Luke Eliot Hodgson4,5
  1. 1 Emergency Department, Western Sussex Hospitals NHS Foundation Trust, Worthing Hospital, Worthing, UK
  2. 2 Patient First Kaizen Improvement, Western Sussex Hospitals NHS Foundation Trust, Worthing, UK
  3. 3 Department of Medicine for the Elderly, Western Sussex Hospitals NHS Foundation Trust, Worthing, UK
  4. 4 Anaesthetics Department, Western Sussex Hospitals NHS Foundation Trust, Worthing, UK
  5. 5 Faculty of Health and Medical Sciences, Department of Clinical and Experimental Medicine, University of Surrey, Guildford, UK
  1. Correspondence to Dr William Shrier, Emergency Department, Western Sussex Hospitals NHS Foundation Trust, Worthing Hospital, Worthing BN11 2DH, West Sussex, UK; willshrier{at}gmail.com

Abstract

Aim To determine the agreement and predictive value of emergency department (ED) triage nurse scoring of frailty using the Rockwood Clinical Frailty Scale (CFS) when compared with inpatient medical assessment using the same scale.

Methods Prospective, dual-centre UK-based study over a 1-year period (1 April 2017 to 31 March 2018) of CFS recorded digitally at nursing triage on ED arrival and on hospital admission by a medical doctor. Inclusion criteria were emergency medical admission in those aged ≥65 staying at least one night in hospital with a CFS completed in both ED and at hospital admission. Agreement between ED triage nurse and inpatient hospital physician was assessed using a weighted Kappa statistic and Spearman’s correlation coefficient. The ability of the ED to diagnose frailty (defined by a CFS ≥5) was assessed using sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and receiver operating characteristic (ROC) curves. At both time points the ability of the CFS to predict inpatient mortality was also assessed.

Results From 29 211 admissions aged ≥65 who stayed at least one night in hospital, 12 385 (42.3%) were referred from the ED. Of the ED referrals, 8568 cases (69.2%) were included with paired CFS performed. Median age was 84 (IQR 77 to 89) with an inpatient mortality of 6%. Median CFS in ED was 4 (3 to 5) and on hospital admission 5 (4 to 6). Agreement between the ED CFS and admission CFS was weak (Kappa 0.21, 95% CI 0.19 to 0.22, rs 0.366). The area under the ROC curve (AUC) was 0.67 (95% CI 0.66 to 0.68) for the ED CFS ability to predict an admission CFS ≥5. To predict inpatient mortality the ED CFS AUC was 0.56 (0.53 to 0.59) and admission CFS AUC 0.70 (0.68 to 0.73).

Conclusion Agreement between ED CFS and inpatient CFS was found to be weak. In addition the ability of ED CFS to predict clinically important outcomes was limited. NPV and PPV for ED CFS cut-off value of ≥5 were found to be low. Further work is required on the feasibility, clinical impact and appropriate tools for screening of frailty in EDs.

  • frailty
  • emergency department
  • triage

Data availability statement

No data are available.

Statistics from Altmetric.com

Key messages

What is already known?

  • The prevalence of frailty is high and is predicted to continue to increase.

  • Frailty is associated with multiple adverse health outcomes and its identification is thought to be crucial to delivering optimal holistic care.

  • The emergency department (ED) is an important step in a patient’s journey at which point frailty must be understood and identified, regardless of subsequent admission or discharge.

What this study adds?

  • Using the simple, pragmatic Clinical Frailty Scale (CFS) at ED triage frailty can be routinely assessed and recorded by the nursing team.

  • Agreement between ED CFS and inpatient CFS was found to be weak.

  • The ability of ED CFS to predict clinically important outcomes was limited.

Introduction

In the UK in 2016 18% of the total population was aged ≥65, but accounted for 24% of emergency department (ED) attendances and 47% of hospital admissions.1 2 Frailty is considered linked though separate from ageing.3 A universally accepted definition of frailty is lacking, however, it has been described as a clinical syndrome of multiple aetiologies, consequent to a cumulative decline in multiple body systems, resulting in decreased physical and cognitive reserve and hence increased vulnerability to stressors.4

From a healthcare systems’ perspective frailty correlates with increased use of resources and higher healthcare expenditure.5 It has been shown that frailty screening in the ED predicts risk of admission, prolonged length of stay (LOS), mortality, nursing home admission and predicts adverse outcomes among those discharged directly from ED.6 It has been postulated that if progress is to be made in the management of the increasing demands on EDs of frail older patients then rapid recognition and response systems are required.7 To expedite access to patient-appropriate pathways and targeted interventions (such as frailty intervention teams in the ED), it is essential to develop tools which can reliably recognise frailty syndromes as early in the ED admission process as possible. A robust easy to use clinical frailty recognition tool could expedite fast track referral to frailty services either within ED or colocated. The multidisciplinary document ‘Quality care for older people with urgent and emergency care needs’ (the Silver Book) recommended that EDs should be configured in such a way that they can screen for common frailty syndromes.8

Several frailty assessment scales have been developed, validated and appraised for their feasibility of use in the ED.9 10 The Clinical Frailty Scale (CFS) was developed by Rockwood and colleagues and has emerged as one of the leading assessments.11 It is an ordinal scale of 9 points, ranging from 1 (very fit) to 9 (terminally ill). It is quick to perform, simple, assesses a variety of variables and does not require specialist equipment.9 12 Internationally the most widely used tool to assess frailty by geriatric clinicians is the gait speed test (43.8%), with CFS the second most used in daily practice (34.3%).13 The ‘Same-day acute frailty services’ document has highlighted CFS as a potentially useful screening tool for frailty.14 The first aim of this study was to determine the agreement of ED triage nurse scoring of frailty using the CFS when compared with a reference standard of inpatient medical assessment using the same scale. Second the study looked to assess the level of agreement between the two CFS assessments across a range of clinically important outcomes (in-hospital mortality, LOS and 30-day hospital readmission). In recognition of the principle function of ED CFS as a screening tool additional analysis was performed to determine the ability of ED CFS to predict equivalent inpatient CFS. For clinically important outcomes patients were divided into two groups represented by CFS of ≥5, with scores of ≥5 representing those patients classified as mild to severely frail. For comparison, analysis was also performed for CFS of ≥4 (vulnerable, mild, moderately and severely frail) as this group has been recognised as the target population for same-day acute frailty services.14

Methods

Setting

Western Sussex Hospitals NHS Foundation Trust is a non-specialist hospital organisation of 870 beds, with two acute sites and a combined annual ED attendance over 135 000. All patients aged ≥65 who attend the ED are expected to have a CFS recorded digitally (Sema-Helix, Atos, France) by the triage nurse as part of their initial assessment (typically within 15 min of arrival). In total, the department has 61 nurses trained in triage who would have contributed scores and 88 doctors on the on-call rota (general and geriatric medicine) during the study period. Printed posters of the CFS visual patient representations and text criteria are located by the triage computers and the ED nurses received a 1-hour training session on the scoring system by a Geriatric Consultant prior to implementation. ED nurses and inpatient doctors were instructed to measure CFS as the functional level 2 weeks prior to acute attendance. All patients aged ≥65 who are admitted to hospital under the medical teams are expected to have a separate CFS recorded digitally (Patientrack Sydney, NSW, Australia) by the ward doctor as part of the initial clerking in an independent frailty assessment. The software interface displays the CFS visual representations and text criteria. The Geriatric Department gives teaching on the CFS as part of their junior doctor induction programme and on-going feedback of scoring at their daily morning handover of new patients. Both the triage in ED and ward scoring are standard practice within both acute sites.

Inclusion criteria

Inclusion criteria were emergency medical admission in those aged ≥65 staying at least one night in hospital with a CFS completed in both ED and at hospital admission.

Study procedures

For the purposes of the study the CFS recorded by the inpatient medical teams was considered the reference standard as they are able to spend more time with the patient, collecting collateral histories, have access to more patient information and have greater expertise in frailty and its assessment. The time between ED and admission frailty assessments was within hours (during the study period over 95% of patients were discharged or admitted from ED within a 4-hour timeframe). There were no formal replicate observations organised and the crossing of raters between the ED and inpatient teams was not possible. In addition to the CFS demographics, LOS, inpatient mortality, 30-day readmissions, International Classification of Diseases (ICD)-10 coded history (congestive cardiac failure (CCF), liver disease, diabetes, chronic kidney disease (CKD, defined as a median estimated glomerular filtration rate (eGFR) <60 mL/min),15 admission National Early Warning Score (NEWS) and the malnutrition universal screening tool (MUST) score were collected. The trust data analyst team pulled the additional data that is routinely collected electronically but was not visible in a live fashion to the clinical team—this collection of data had no impact on the way clinical care was delivered. The study prospectively collected data from 1 April 2017 to 31 March 2018. Patients were followed up for the duration of their inpatient hospital stay and we were subsequently able to link whether the patients were readmitted within the following 30 days after discharge. Follow-up for readmission data (within 30 days) was collected, from routinely available electronically stored information by the Trust data analysts with no link to the research team. The inpatient assessors were blinded to the ED CFS as they do not have access to the section of software where the ED scores were recorded. Data analysts from the Trust not involved with the research team extracted the data electronically from the hospital server and anonymised the dataset. None of the researchers involved in data analysis were involved in the management of the patients, they had access only to the fully anonymised individual level data and were blinded to other patient data, such as clinical notes, imaging or diagnoses on admission.

Follow-up

Patients were followed up for the duration of their inpatient hospital stay and we were subsequently able to link whether the patients were readmitted within the following 30 days after discharge.

Outcomes

The primary outcome of interest was agreement of the CFS, across its ordinal scale, between ED triage nurses and the inpatient medical team. The secondary outcomes assessed were a comparison between the diagnostic accuracy (AUC) of both CFS scores for in-hospital mortality, LOS and 30-day hospital readmission. In addition sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive and negative likelihood ratio with 95% CIs were used to estimate the accuracy of ED Triage CFS to predict inpatient CFS.

Statistical analyses

Clinically, the magnitude of difference in CFS is of significance. For example, a disagreement of 1 point between a score of 8 versus 9 is of less significance than a disagreement of 8 points (eg, 1 vs 9). Cohen’s weighted Kappa statistic (linear weighting using SPSS V.25) was therefore used with continuous sampling to assess agreement between the ED and inpatient teams. In a 1981 simulation study Cicchetti suggested for ordinal scales with nine categories of classification a sample size of 160 would be sufficient.16 Landis and Koch proposed standards for strength of agreement for the Kappa coefficient as: ≤0 = poor, 0.01–0.20=slight, 0.21–0.40=fair, 0.41–0.60=moderate, 0.61–0.80 substantial and 0.81–1=almost perfect.17 Sensitivity, specificity, PPV, NPV, positive and negative likelihood ratio with 95% CIs were used to estimate the accuracy of ED Triage CFS of ≥5 to predict inpatient CFS of ≥5. These analyses were repeated for ED CFS of ≥4 to predict inpatient CFS of ≥4. Area under the receiver operating characteristic (AUROC) curves were plotted to assess the ED CFS ability to predict both an admission CFS of ≥4 and≥5, and the ability of both ED and admission scores to predict in-hospital mortality. Logistic regression (backward stepwise elimination) was applied to investigate independent predictors of inpatient mortality first including ED CFS and second the admission CFS with results presented as ORs with 95% CIs. Backward elimination was employed which can be particularly useful when collinearity is present.18 In addition to CFS potential predictors were age, gender, CCF, liver disease, diabetes, CKD, community acute kidney injury (KDIGO increase 1.5 baseline creatinine)19 NEWS and MUST. Spearman’s rank correlation coefficient, appropriate when one or both variables are ordinal was used to compare correlation of the two CFS ratings with LOS.20 Using the Fisher r-to-z transformation, a value of z was calculated to assess the significance of the difference between two correlation coefficients (online calculator: http://vassarstats.net/rdiff.html). Data were extracted fully anonymised on Microsoft Excel and all analyses were conducted using SPSS IBM (V.25).

Results

Over the study period there were 29 211 admissions aged ≥65 (both ED and direct admissions to specialty teams) who stayed at least one night in hospital. Of these 12 385 (42.3%) were referred to inpatient teams from the ED, of whom 8568 (69.2%) had both an ED triage and medical CFS available for analysis (see figure 1).

Figure 1

Flowchart of study participants. CFS, Clinical Frailty Scale.

The overall median age was 82 (IQR 74–88). In those without paired frailty assessment median age was 81 (IQR 74–87) and with a paired assessment 84 (IQR 77–89). In the cohort with paired assessments 54% were male, LOS was 9 (IQR 5 to 18) days and inpatient mortality 6% (n=474, see table 1 for demographics). About 23% of admissions were followed by a readmission within 30 days.

Table 1

Demographics, admission information frailty scores and past medical history within the cohort

Median CFS in ED was 4 (IQR 3 to 5) and on hospital admission 5 (IQR 4 to 6), of those with paired ratings 34% of the ED and 60% of the admission CFS assessments were ≥5 (59% ED and 79% admissions ≥4). Table 2 shows the cross-tabulation of frailty score categories assigned by the ED compared with the teams with 21% perfect agreement, 57% within 2 points and 21% >2 points. Overall n=4801 (56%) of the pairs were within ±1 point of each other. A relatively small number of cases had large disagreements—2.5% (n=214) by 5 points or more, with 93% representing a lower score by the ED team.

Table 2

Cross-tabulation of ED versus admission CFS

The weighted kappa was 0.21 (95% CI 0.19 to 0.22) for agreement between the two CFS assessments. Spearman’s correlation coefficient was 0.366 (p<0.0001). ED CFS of ≥5 performed better than ED CFS of ≥4 for specificity (79.89%, 95% CI 78.51% to 81.22% and 66.44%, 95% CI 64.21% to 68.63%) and NPV (48.86%, 95% CI 48.12% to 46.59% and 33.65%, 95% CI 32.63% to 34.69%), but had lower sensitivity and PPV (table 3). For the ED score (across the CFS scale 1–9) to predict a CFS of ≥5 on admission the area under the ROC curve (AUC) was 0.67 (0.66 to 0.68, see figure 2) and 0.70 (0.69–0.71) for CFS of ≥4. To predict inpatient mortality the admission frailty score had a higher AUC (0.70 (0.68 to 0.73)) than the ED CFS (0.56 (0.53 to 0.59))—see figure 3. The CFS on hospital admission had a higher AUC than the NEWS on admission to the inpatient ward (0.68 (0.65 to 0.70)) to predict mortality.

Figure 2

AUC for ED CFS (blue line) to predict an admission CFS 5+0.67 (95% CI 0.66 to 0.68). AUC, area under the ROC curve; CFS, Clinical Frailty Scale; ED, emergency department; ROC, receiver operating characteristic curve.

Figure 3

AUCs to predict inpatient mortality. Admission CFS (blue line): 0.70 (95% CI 0.68 to 0.73), ED CFS (red line): 0.56 (0.53 to 0.59). AUC, area under the ROC curve; ROC, receiver operating characteristic curve.

Table 3

Diagnostic performance of ED CFS ≥4 and ≥5 to predict inpatient CFS ≥4 and ≥5

Both ED and admission CFS ≥5 were associated with a LOS ≥7 days though neither CFS rating had high discrimination on AUC analysis (table 4). Correlation of CFS with LOS was higher for inpatient assessment than ED triage, although poor for both ratings (for ED 0.13 and 0.26 for admission, p<0.001). Neither CFS predicted 30-day hospital readmission (see online supplemental appendix for AUC curves). On logistic regression admission CFS (OR 1.51 (95% CI 1.40 to 1.63), was an independent predictor of mortality alongside community AKI, MUST score, NEWS, female sex and age. ED CFS was not an independent predictor on a separate regression analysis using the same variables. Relationship between CFS assessment scores (admission and in ED) with age, LOS and inpatient mortality across scores 1–9 are displayed in table 5.

Supplemental material

Table 4

Relationship between CFS and LOS, inpatient mortality and 30-day readmission

Table 5

Relationship between CFS assessment scores (admission and in ED) with age and LOS and inpatient mortality

Discussion

This dual-centre UK study is the first to examine agreement of the CFS between ED triage and inpatient medical teams. While this study has demonstrated that it is possible to screen for frailty in the ED, it is important to note that agreement between screening at triage and subsequent inpatient assessment was weak (kappa 0.21). The kappa should be interpreted with caution as both the time constraints on ED triage and the relative lack of collateral information would be anticipated to have an adverse effect on the reproducibility of test conditions between ED and the inpatient environment. Correlation of the two datasets was however also found to be low positive (rs 0.366). Previous small studies (30–159 patients) specifically looking at the CFS reported good agreement. A Canadian study assessing frailty in patients unable to attend a geriatric clinic, and a UK study assessing frailty in patients admitted to critical care, both found a kappa of 0.64 for the inter-rater reliability of the CFS as assessed by a medical student and geriatric nurse/critical care doctor, respectively. An American study assessing frailty in a vascular outpatient clinic found a kappa of 0.79 between the CFS produced by a medical assistant and vascular surgeon.21–23 Frailty is associated with adverse health outcomes and increased LOS in hospital, hence its identification and understanding is important in multiple healthcare settings. In the ED recognition may inform clinical decisions, among those patients being admitted to hospital and those under consideration for discharge. In this study, we found the predictive value of CFS in ED for LOS, 30-day hospital readmission or mortality to be limited (AUC of 0.56, 0.51 and 0.56, respectively). The predictive value for inpatient mortality of CFS on admission was 0.7 and consistent with previous studies of inpatient CFS on emergency admissions to specialist geriatric wards (AUC 0.72 and 0.74).24 25 CFS has been proposed as a potential screening tool for frailty within ED14 and in this context CFS of ≥4 showed better sensitivity and NPV while CFS of ≥5 was found to be more specific. Overall sensitivity was found to be low particularly for CFS of ≥5 (43.61%), suggesting that CFS at ED triage has limited potential for screening and resource allocation of more specialised CGA. (table 3). The setting of this study is of particular interest as the current percentage of the study population aged ≥85 is similar to the national projections for 2041.26 27 Despite a high number of patients attending ED aged >65 years the capture rate of CFS for all attendances to ED was 80% and of admissions only 69.2% had paired CFS. The capture rate was similar to previous studies of all non-elective inpatient episodes from ED admission to discharge (71.2%–76.5%).24 25

There are several potential explanations for the low agreement and limited ability of the CFS to predict outcomes. Agreement was between two teams each of which had multiple members with varying levels of clinical experience. Although the CFS is relatively quick to assess, in an already stretched ED, it may not be deemed possible to allocate the time needed to accurately assess frailty in detail required by the CFS. Additional factors intrinsic to the ED which make ascertainment of pertinent information difficult including an environment with limited privacy, and the acute nature of presentations which may limit a patient’s ability to communicate and staff’s ability to gain an accurate collateral history. The training period may have been insufficient and the lack of immediate clinical application may have impaired stakeholder engagement. Median ED CFS was found to be 4 compared with 5 for admission CFS (table 1), the possibility exists that a lack of confidence in applying the tool and fear of incorrectly labelling patients frail resulted in under scoring at ED triage. It is possible that the CFS tool is not the most appropriate tool for frailty screening in ED particularly in the time pressured context of triage, although it is important to consider the time delays inherent in other members of clinical staff (eg, bedside nurse/medical staff) initiating screening.

Limitations

This study limited assessment of frailty to only the CFS while several other tools have been developed, validated and shown to be feasible for use in the ED. Although the two assessments were carried out at different time points, this would have only been a matter of a few hours. However, the extra information available to the admitting team over this time may have allowed a more accurate appraisal to be made. Review of inpatient CFS by an geriatric consultant was standard practice in both study sites, however not all patients were admitted to geriatric wards resulting in non-specialist review with subsequent impact on the reference standard. The possibility that ED CFS scores could have been written in clinical notes exists resulting in potential bias; however, this was not routine clinical practice during the study period in view of the existing electronic capture. Including assessments by multiple different raters at two time points could be argued as a limitation. However the CFS has been employed to be a pragmatic tool that like the NEWS could allow a common language between teams as a frailty screen and thus assessing multiple raters can be justified. Standard practice nationally and at the study site is to use the functional level 2 weeks prior to acute attendance. As a consequence there should be no impact on correlation secondary to delay between ED and ward scoring. The demographic of the population in which this study took place is characterised by a high prevalence of advanced age and frailty, which may limit the generalisability of the presented results. However, as outlined, most societies are experiencing a growing geriatric and frail demographic. A relatively small number of cases disagreed to a large extent (2.5% by ≥5 points) and future studies could investigate why, as correct application of frailty assessment should reflect a patient’s baseline state prior to the acute event.

Future work

Qualitative work exploring the possible factors contributing to underscoring at ED triage could give potentially valuable insights into future interventions to improve agreement and clinical usefulness of the CFS within ED. Beyond identification of frailty, exploration might be indicated into whether the ED assessment of frailty is sufficient to identify a cohort that may benefit from a specialist geriatric multidisciplinary team working within the ED. Such a service could provide proactive comprehensive geriatric assessments with a variety of potential benefits including reduced reattendances and facilitation of early safe discharge due to their expertise in the field and knowledge of appropriate community services. This has the potential to reduce hospital admissions and its attendant significant associated risks for this vulnerable group. Previous research suggested such assessments for older people in general could reduce conversion to hospital admission though at the time other age groups not experiencing the intervention were also less likely to be admitted and the study did not focus only on frail groups.28

Conclusion

Agreement between ED CFS and inpatient CFS was found to be weak. The ability of ED CFS to predict clinically important outcomes was limited and sensitivity was found to be low particularly for CFS of ≥5 (43.61%). Further work is required on the feasibility, clinical impact and appropriate tools for screening of frailty in EDs.

Data availability statement

No data are available.

Ethics statements

Patient consent for publication

Ethics approval

Ethical approval was given by NHS South Central—Hampshire B Research Ethics Committee (REC reference 18/SC/0513).

References

Footnotes

  • Handling editor Mary Dawood

  • Contributors WS, CD and LEH were involved in writing the manuscript. CD, LEH and DH were involved in the planning and conduct of the study. WS was involved in submitting the manuscript. PP and LEH were involved in the statistical analysis. DH was involved in the geriatric training session. WS and CD were responsible for the overall content.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.