Table 1

Sources of rater error and potential solutions

Simple mistakes made with categorisation of behavioursThese errors would be reduced by improved training in use of the tool, or additional behavioural markers to better define each skill.
Overlapping definitions of a skill
Misunderstandings of the definition of a skill
Concealed cognitive processes and reasoningThis is a weakness of any behavioural marker system. However, this can be mitigated by good debriefing, where registrars have an opportunity to discuss the reasoning behind observed behaviour.
Failure to rate objectivelyObservers may benefit from receiving more advanced training on sources of bias and cognitive error, such as leniency/severity and the halo/horn effect.
Missed behavioursWith any behavioural marker system, the observer is vulnerable to making simple mistakes, may miss some behaviours or may fail to comment on some observed behaviours. This is particularly so with a tool used in the ED where observers are subject to high volumes of information and may suffer observer fatigue.
Observed behaviour not recorded/judged
Disagreement between observers regarding what constitutes good and poor behaviourDisagreements between observers regarding what constitutes good and poor behaviour is likely to improve as EM physicians receive more training in non-technical skills in general.
Observer not familiar with the departmentThese errors are a consequence of the research process and would not happen during a true workplace-based assessment where assessors would be familiar with the team and working environment.