Jump to comment:

• Interrater-variability in frailty screening using the Safety Management System (VMS).
  Wendy van der Heiden
  Published on: 14 April 2021

• Published on: 14 April 2021

  Interrater-variability in frailty screening using the Safety Management System (VMS).

  • Wendy van der Heiden, MD Ziekenhuis Amstelland

   

  To the editor,

  In their article “Agreement and predictive value of the Rockwood Clinical Frailty Scale (CFS) at emergency department triage”, Shrier et al (1) nicely illustrate that the level of agreement between different health care professionals in obtaining the CFS at different clinical settings is weak. The CFS was rated for 8,568 patients over 65 years by the triage nurse at the emergency department (ED) and by the attending physician on admission on the ward. Both scores were compared using the Cohen’s kappa coefficient, which was  0.21 and therefore is considered weak (2). 
  We found similar results in our AmsterGEM study. The AmsterGEM study is an observational prospective cohort study that investigates the prognostic accuracy of frequently used frailty screening instruments (3). Patients aged over 70 years old attending the ED were screened with four screening instruments, including the Safety Management System (VMS) (4). The VMS consists of four geriatric domains, which are associated with functional decline: delirium, malnutrition, falls and ADL limitations (5,6). All Dutch hospitals have implemented the VMS screening instrument for hospitalized older patients. In a sub-analysis of our study, the VMS was applied on 173 hospitalized patients over 70 years old, at the ED by a research student and at admission by the attending nurse on the ward. The average age was 81.2 years old, 81 (47%) were male and 111 s...

  Show More

   

  To the editor,

  In their article “Agreement and predictive value of the Rockwood Clinical Frailty Scale (CFS) at emergency department triage”, Shrier et al (1) nicely illustrate that the level of agreement between different health care professionals in obtaining the CFS at different clinical settings is weak. The CFS was rated for 8,568 patients over 65 years by the triage nurse at the emergency department (ED) and by the attending physician on admission on the ward. Both scores were compared using the Cohen’s kappa coefficient, which was  0.21 and therefore is considered weak (2). 
  We found similar results in our AmsterGEM study. The AmsterGEM study is an observational prospective cohort study that investigates the prognostic accuracy of frequently used frailty screening instruments (3). Patients aged over 70 years old attending the ED were screened with four screening instruments, including the Safety Management System (VMS) (4). The VMS consists of four geriatric domains, which are associated with functional decline: delirium, malnutrition, falls and ADL limitations (5,6). All Dutch hospitals have implemented the VMS screening instrument for hospitalized older patients. In a sub-analysis of our study, the VMS was applied on 173 hospitalized patients over 70 years old, at the ED by a research student and at admission by the attending nurse on the ward. The average age was 81.2 years old, 81 (47%) were male and 111 scored a positive VMS by the nurse and 132 scored a positive VMS by the student. The level of agreement evaluated by Cohen’s kappa coefficient was weak (0.25),  as shown in table 1. The domain 'falls' showed the highest agreement, with an acceptable kappa coefficient of 0.61. This could be explained by the fact that questions concerning increased fall risk were asked strictly as written in the VMS. In addition, the percentage of patients with a positive frailty score was higher for students than for nurses (36% vs 24%). Nurses stated they often relied on their own impression of the patients when filling in the questions, where students asked the questions literally. Older patients with memory problems or the clinical setting are other reasons for the low level of agreement. The ED is a busy and hastily place for the patient, caregiver and healthcare professional, which may complicate frailty scoring.
  In conclusion, the awareness of the increased risk of negative health outcomes in frail older patients is an important reason why frailty screening should be part of the initial evaluation of older patients. However, the low inter-rater reliability contributes to moderate clinical accuracy of frailty instruments. Hence, training of staffs who apply screening instruments is necessary to minimalize interrater-variability. 

  Sincerely,

  W. van der Heiden, MD
  C.S. van Dam, MD
  M.J.L. Peters, MD PhD
  M.C. Trappenburg, MD PhD 

  Department of Internal Medicine
  Ziekenhuis Amstelland, Amstelveen, the Netherlands
  
  Department of Internal Medicine and Geriatrics
  Amsterdam UMC, location VUmc, Amsterdam, the Netherlands

   

  Table 1. Positive VMS scores per domain by student and nurse, with corresponding kappa.

  	Student	Nurse	Kappa
  Delirium	81	43	0.23
  Falls	73	53	0.61
  Malnutrition	89	77	0.18
  ADL limitations	73	95	0.21
  Total VMS	132	111	0.25

   

  References

  1.              Shrier W, Dewar C, Parrella P, Hunt D, Hodgson LE. Agreement and predictive value of the Rockwood Clinical Frailty Scale at emergency  department triage. Emerg Med J. 2020 Nov;

  2.              Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977 Mar;33(1):159–74.

  3.              C.S. van Dam, M.C. Trappenburg, M.M. ter Wee, E.O. Hoogendijk, H.C.W. de Vet, Y.M. Smulders, P.W.B. Nanayakkara, M. Muller MJLP. The accuracy of four frequently used frailty instruments for the prediction of adverse health outcomes among older adults at two Dutch Emergency Departments: findings of the AmsterGEM study. Manuscript submitted for publication.

  4.              van Noord I, Zwijnenberg N, Wagner C. Patiëntveiligheidscultuur in Nederlandse Ziekenhuizen: Een stap in de goede richting. 2013. 44 p.

  5.              Heim N, van Fenema EM, Weverling-Rijnsburger AWE, Tuijl JP, Jue P, Oleksik AM, et al. Optimal screening for increased risk for adverse outcomes in hospitalised older  adults. Age Ageing. 2015 Mar;44(2):239–44.

  6.              Heim N. Kwetsbare patiënten opsporen met VMS. Med Contact (Bussum). 2013;68(2):94–6.

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top