Sources of rater error and potential solutions
| Error | Comment/solution |
| Simple mistakes made with categorisation of behaviours | These errors would be reduced by improved training in use of the tool, or additional behavioural markers to better define each skill. |
| Overlapping definitions of a skill | |
| Misunderstandings of the definition of a skill | |
| Concealed cognitive processes and reasoning | This is a weakness of any behavioural marker system. However, this can be mitigated by good debriefing, where registrars have an opportunity to discuss the reasoning behind observed behaviour. |
| Failure to rate objectively | Observers may benefit from receiving more advanced training on sources of bias and cognitive error, such as leniency/severity and the halo/horn effect. |
| Missed behaviours | With any behavioural marker system, the observer is vulnerable to making simple mistakes, may miss some behaviours or may fail to comment on some observed behaviours. This is particularly so with a tool used in the ED where observers are subject to high volumes of information and may suffer observer fatigue. |
| Observed behaviour not recorded/judged | |
| Disagreement between observers regarding what constitutes good and poor behaviour | Disagreements between observers regarding what constitutes good and poor behaviour is likely to improve as EM physicians receive more training in non-technical skills in general. |
| Observer not familiar with the department | These errors are a consequence of the research process and would not happen during a true workplace-based assessment where assessors would be familiar with the team and working environment. |