Original article
The analysis of ordinal agreement data: beyond weighted kappa

https://doi.org/10.1016/0895-4356(93)90173-XGet rights and content

Abstract

The weighted kappa statistic has been used as an agreement index for ordinal data. Using data on the comparability of primary and proxy respondent reports of alcohol drinking frequency we show that the value of weighted kappa can be sensitive to the choice of weights. The distinction between association and agreement is clarified and it is shown that in some respects weighted kappa behaves more like a measure of association than an index of agreement. In particular, it is demonstrated that the weighted kappa statistic is not always sensitive to differences in the observed proportion in exact agreement and that high values of weighted kappa can be observed even when the level of agreement is low. We illustrate the use of statistical models in the analysis of epidemiologic agreement data and conclude that modelling ordinal agreement data produces insights which cannot be obtained through the use of weighted kappa statistics.

References (23)

  • A. Agresti

    A model for agreement between raters on an ordinal scale

    Biometrics

    (1988)
  • Cited by (95)

    • Item desirability matching in forced-choice test construction

      2021, Personality and Individual Differences
    • Summary measures of agreement and association between many raters' ordinal classifications

      2017, Annals of Epidemiology
      Citation Excerpt :

      For more detail, see Shrout and Fleiss [5]. The ICC has been demonstrated to be equivalent to Cohen's weighted kappa with quadratic weights for the case of two raters [3,28,26]. The ICC can be easily calculated using most statistical software packages including the irr package in R and the official INTRACC macro in SAS.

    • Reply to the Letter by Siamak Sabour, Fariba Ghassemi

      2016, Revue de Chirurgie Orthopedique et Traumatologique
    • Reply to the Letter by Siamak Sabour, Fariba Ghassemi

      2016, Orthopaedics and Traumatology: Surgery and Research
    View all citing articles on Scopus
    View full text