Background Delayed diagnoses of serious emergency conditions can lead to morbidity in children, but are challenging to identify and measure. We developed and piloted an automated tool for identifying delayed diagnosis of two serious conditions commonly seen in the ED using administrative data.

Methods We identified cases with a final diagnosis of appendicitis or sepsis in a freestanding children’s hospital from 2008 to 2018, with any hospital ED encounter within the preceding 7 days. Two investigators reviewed a subset of these cases using the electronic health records (EHR) to determine if there was a delayed diagnosis and interrater reliability was assessed using the intraclass correlation coefficient (ICC). An automated tool was applied to the same cases to assess its positive predictive value (PPV) to identify those with a delayed diagnosis, using the manual chart review as the gold standard. The tool used number of days since visit, presence of a related diagnosis on the initial visit, and whether or not the patient was discharged.

Results Previous ED encounters preceded 91/3703 (2.5%) appendicitis cases and 159/1754 (9.1%) sepsis cases; 78 cases of each were sampled for review. In manual review, 73.4% and 22.8% were thought to have delayed diagnoses; reviewer agreement was excellent (appendicitis ICC 0.77, 95% CI 0.62 to 0.86 and sepsis ICC 0.77, 95% CI 0.43 to 0.89). The PPVs of the automated tool for determination of delayed diagnosis for appendicitis within 1, 3 or 7 days were 96.2%, 95.1% and 93.6%, respectively. For sepsis, the PPVs were 71.4%, 63.6% and 41.2% within 1, 3 or 7 days, respectively.

Conclusions This automated tool performed well compared with expert EHR review. Performance was stronger for appendicitis. Further tool refinement could improve performance.