Objective Inter-rater reliability (IRR) is rarely determined for scoring systems used to recognise deterioration in children. Thus, the primary objective of this study was to determine the IRR of the Paediatric Observation Priority Score (POPS), a bespoke paediatric scoring system for ED use. The IRR of both the overall POPS and its individual parameters are to be investigated.

Methods This is an experimental, single-centre study based in the Leicester children’s ED, England. A purposive sample of nursing, healthcare assistants and doctors were recruited from October 2017 to December 2017. Eleven prerecorded video assessments were shown to 45 participants asked to generate a POPS for each child. The participants were blinded to each other’s scores and the triage POPS. IRR among the participants POPS was measured using the intraclass correlation coefficient (ICC). Fleiss kappa was used to determine the IRR of individual parameters.

Results Overall, the ICC across all assessments by staff was 0.84 (95% CI 0.71 to 0.95). Fleiss kappa for parameters ranged from 0.87 to 1 for oxygen saturation, 0.48 to 0.91 for work of breathing, 0.55 to 1 for response, 0.46 to 0.87 for gut feeling, 0.53 to 1 for medical history, 0.76 to 1 for heart rate, 0.44 to 0.96 for respiratory rate and 0.51 to 1 for temperature.

Conclusions This study suggests there is statistically moderate to excellent IRR of the POPS when assessing a variety of clinical presentations between different healthcare professionals with a range of experience.