Objectives and Background Recently the usefulness of time targets for Emergency Department (ED) length of stay as a quality indicator has been questioned, and there has been a call to use more clinically orientated indicators. However there is uncertainty about which markers should be used and to date indicator selection has been largely by expert opinion. The aim of this study was to develop an objective tool which can be used to critically appraise candidate indicators of quality of care in the ED.

Methods A search of websites of EQUATOR network, CASP, Cochrane Effective Practice, EPIQ, NHS, RAND, TJC and NIH was conducted, along with structured searches in Medline, Embase and CInAHL and communication with experts in the field. Potential tools were compared using an evidence table.

Results Thirty two potential tools were identified. All considered some aspects of the validity and the technical properties of indicators to a lesser or greater extent. All were based on “Level five” evidence. There were differences in terminology and completeness of the tools. Only one was ED specific and none had a closing summary reflecting the final recommendation about whether the indicator under investigation should be used. An “ED Quality Indicator Critical Appraisal Tool” was developed, using elements of the existing tools, with provision to consider the value of the indicator both within the ED and as a marker of whole system quality of care.

Conclusion An “ED Quality Indicator Critical Appraisal Tool” (ED QICAT) has been developed. If validated, this may be used by clinicians and health policy makers to objectively evaluate and compare ED based candidate quality of care indicators prior to their adoption and implementation.