Article Text
PDF
Abstract
Introduction and aims In the UK, specialist trainees in emergency medicine are required to pass the Fellowship of the College of Emergency Medicine (FCEM). This examination assesses clinical knowledge, attitudes and skills, management principles, critical appraisal, and the ability to search medical literature and synthesise information. The aims of this study were to ascertain what resources trainees felt were most valuable in preparation for the FCEM and to obtain trainee feedback on the running of the FCEM.
Methods A questionnaire was developed in conjunction with the TSC into nine parts covering all aspects of preparation for and experience of sitting the FCEM. Email addresses of those trainees who had sat the FCEM examination in 2006 and 2007 were provided by the CEM and questionnaires were sent electronically to recipients. Responses were collated and analysed using Microsoft Excel.
Results There was a response rate of 42% (86/203), of whom about three-quarters felt well prepared for the FCEM. The most highly valued resources for exam preparation were practice questions, private study and small group work. A yearly mock FCEM examination was felt to be important by those who had such access and local trainer involvement in exam preparation was perceived significant for success.
Conclusions Training programmes should make sure that facilities and expertise are available at a local level to allow trainees to have access to everything that is considered important in order to pass the FCEM.
- Education
- emergency care systems
- emergency departments
- training
- assessment
- neurology
- headache
- abdomen non-trauma
- admission avoidance
- anaesthesia
- arrhythmia
- acute myocardial infarct
- acute coronary syndrome
Statistics from Altmetric.com
- Education
- emergency care systems
- emergency departments
- training
- assessment
- neurology
- headache
- abdomen non-trauma
- admission avoidance
- anaesthesia
- arrhythmia
- acute myocardial infarct
- acute coronary syndrome
Introduction and aims
In the UK, specialist trainees in emergency medicine (EM) have been required to sit an exit examination since 1996. In 2006, the College of Emergency Medicine (CEM) was formed and the examination became the Fellowship of the College of Emergency Medicine (FCEM). Since its inception, the exit examination has evolved and developed to keep pace with the rapid development of EM as a specialty. The FCEM represents a significant hurdle to clear as it covers clinical knowledge in the form of a short answer question (SAQ) paper and clinical attitudes and skills in an objective structured clinical examination (OSCE). Critical appraisal (CA) skills are assessed via a written paper (formerly by viva) and understanding of management principles by viva examination. The candidate is also required to submit a clinical topic review (CTR) for viva examination. This is a significant piece of work that tests the ability to formulate a research question, search the relevant medical literature and synthesise the information into pertinent conclusions.
The primary aim of this study was to ascertain what resources trainees felt were most valuable in preparation for the different FCEM components. This study was also authorised by the Training Standards Committee (TSC) of the CEM who added some questions to evaluate trainee feedback on examiner behaviour during the FCEM.
Methods
A questionnaire, developed in conjunction with the TSC, was divided into nine parts encompassing all aspects of preparation for, and experience of, sitting the FCEM examination. Questions 1–3 were designed to assess the respondent's self-assessment of their level of preparedness for the FCEM exam. Questions 4–8 assessed the value of a variety of resources used for preparation for each component of the FCEM. These resources were gleaned from the experience of two authors (JC had recently passed the FCEM, and WH has extensive experience as a trainer and was then chair of the TSC), the CEM website1 and the opinion of the members of the TSC of the CEM. Question 9 was developed by the TSC and split into two parts in order to gain information regarding examiner behaviour in (a) the OSCE and (b) the viva sections of the exam.
For each question, respondents were asked to grade their answers according to a 5-level Likert type scale.2 There was opportunity for free text answers to each section (see online appendix 1 for a complete copy of the questionnaire).
Email addresses of those who had sat for the FCEM examination in 2006 and 2007 were provided from CEM records and questionnaires were sent electronically in February 2008. A follow-up reminder email was sent to non-responders 1 month later. In addition, the questionnaire was piloted on four trainees within the Yorkshire Deanery who had sat the FCEM examination in the autumn of 2005 and these results were included in the analysis. Responses were collated and analysed using Microsoft Excel.
Results
Nineteen of 224 emails sent were returned undeliverable and a response rate of 42% (86/203) was achieved from the remainder. Not all respondents answered all parts of each questions causing the denominator to vary slightly from 83 to 86 between questions. Results have been expressed as total numbers (Questions 1, 2 and 9) and means (Questions 3–8). Where means are given for the responses, those who did not have access to specific items for preparation are not accounted for.
Sixty-six respondents (77%) passed the FCEM at their first attempt and, at the time of completing the questionnaire, 82 (95%) had been successful. Of these, 55 candidates (64%) had completed the FCEM in 2007 and one did so in the Spring of 2008.
Q1 How well prepared did you feel for your first sitting of the FCEM?
Around 75% (73%–77%) expressed that they were well to extensively prepared for each section of the FCEM (table 1).
Candidate preparedness for the FCEM
Q2a When first sitting the FCEM exam was the format as you expected it to be? For instance were the vivas run as you expected, did the SAQ questions follow what you thought they would ask, were the OSCE stations run as you would expect? Answers were given for each section of the FCEM
Over 70% of respondents felt that they had no surprises with the formats of the CTR, OSCE, CA and Management components of the FCEM, though only 53% found the format of the SAQ as expected (table 2).
Format and content of FCEM exam components as expected
Q2b When first sitting the FCEM was the content as you expected? For instance was the balance of clinical work appropriate? Were the elements relevant to your work? Did each section's content meet with your expectations? Answers were given for each section of the FCEM
Over 70% of respondents had no surprises with the content of the CTR, CA and Management vivas though this dropped to 61% for the OSCE and to only 44% for the SAQ (table 2).
For questions 3–8 respondents were asked to give answers on a Likert2 type graded scale: 1 (not important); 2 (slightly useful); 3 (recommended); 4 (very important); and 5 (invaluable).
Q3 In your personal experience how would you rate the following in terms of importance in exam preparation? (Scored 1–5)
Those most highly valued resources for exam preparation overall were practice questions (mean 4.3), private study (mean 4.1) and small group study with peers (mean 3.9). A yearly mock FCEM examination was not available to 22% of respondents in any form but was highly valued (mean 3.9) by those who had such access (figure 1).
Overall value of revision resources for the Fellowship of the College of Emergency Medicine (FCEM). SFP, shop floor practice.
Recurring free text comments included the need for increased dissemination of practical exam guidance from the CEM, including better access to FCEM past paper materials and also practical matters such as the requirement to write in pencil and that calculator use was forbidden. There was concern that many consultant trainers had never sat or examined the FCEM, and that a proportion were ‘disengaged’ and had ‘little idea’ what the exam entailed. The CEM exam days were highlighted as very beneficial in demystifying the FCEM and most found the curriculum3 difficult to use as an exam preparatory tool.
Q4 Importance of the following in CTR preparation (Scored 1–5)
Pre-exam practice CTR vivas were considered the most important preparatory tool (mean 4.2) followed by guidance from local trainers, information on the CEM website1 (mean 3.6) and literature searching training. A yearly CTR viva was not available to 60% of respondents but those who had access to this felt it worthwhile (table 3).
Most valued resources for each component of the FCEM
Free text comments centred on the importance of deciding on a question and securing electronic access to journals at an early stage. Many highlighted the benefit of producing a CTR well in advance (at least 6 months) of the exam so as to be able to focus on the other parts of the FCEM. FCEM revision courses were thought to be generally of a high quality for this section of the exam. Some questioned the relevance of the CTR to work as an EM consultant with reports of it being ‘time-consuming’ and ‘worthless’. However, the majority saw it as a valuable exercise that allowed them to develop skills that they otherwise would not have. Indeed, frustration was often expressed that the examiners had not read the CTRs in enough detail and many felt that a 15 min viva was inadequate to assess such a significant piece of work.
Q5 Importance of the following in critical appraisal preparation (Scored 1–5)i
Mock CA exam practice (mean 4.3) was considered the principal tool in preparation for this section of the exam, though specific CA sessions during training were also highly regarded (mean 4.0), followed by small group work and attendance at a formal CA course (table 3). A regular departmental journal club was unavailable to 25% of trainees and only highly valued by 33% of those that had access. The advice given on the CEM website1 regarding this section of the exam was near the bottom of those resources surveyed (mean of 3.1).
Comments in the free text were to ‘practise, practise, practise’ and that journal clubs needed to be disciplined and organised to be of benefit. Books such as those by Sackett et al 4 and Greenhalgh5 were often used and some praised the Diploma in Health Services Research.
Q6 Importance of the following in management viva preparation (Scored 1–5)
By far the most valued method of preparation for this section of the exam was participation in mock examination in in-tray exercises (mean 4.5). Also important was local advice from trainers (mean 3.9), regular management discussions with consultants and small group revision (table 3). Management courses ranged from ‘disappointing’ to ‘excellent’ but were generally regarded as being ‘useful but unhelpful for the exam.’ The CEM guidance1 for this section scored a mean of 2.8. The free text highlighted the Medicolegal Pocketbook6 as being a valuable resource and that information from NHS, BMA, GMC, DH and Defence Union websites was often useful.
Q7 Importance of the following in OSCE preparation (Scored 1–5)
The practice of OSCE scenarios was given the highest weighting in preparation for this section (mean 4.5). Small group practice (mean 4.2), a yearly OSCE assessment and the production of a list of OSCE templates also scored highly (table 3). The CEM advice1 for this section scored a mean of 3.3.
The free text comments emphasised the need to practice OSCE technique for a variety of scenarios (procedures, history taking, examination, teaching, etc) at every opportunity and to be clearly aware of what was being assessed in each station. Many expressed that stations were often ‘false and contrived’ but practise was key to success and knowledge of life support algorithms essential for ‘easy marks’.
Q8 Importance of the following in SAQ preparation (Scored 1–5)
Using practice SAQ papers (mean 4.4) and SAQ questions from other trainees (mean 4.1) were thought the most important methods of revising for this aspect of the FCEM followed by a yearly mock exam (table 3). The free text emphasised the importance of familiarity with SAQ question format and timing and that it was also important to learn guidelines and be abreast of current topics in the specialty.
There was a general feeling that more FCEM SAQs should be made available through the CEM1 and that guidance on the format and marking could be clearer. Some stated that questions on guidelines and scoring systems may not be necessary, as in clinical practice one can look them up. Most accepted that, while technique was important for SAQ success, ‘hard graft’ with EM textbooks and other resources was required to lay the groundwork.
Q9a What is your experience of the examiner behaviour in the OSCE?
Answers to this question demonstrated that examiner conduct was generally favourable in all aspects of the OSCE (table 4). The only points of note were that 50% of candidates felt the examiners were difficult to read and 38% felt that further useful information could have been provided as the scenarios progressed. In the free text, several respondents felt that some scenarios were presented in an inflexible manner and that priorities could be unclear.
Examiner behaviour in the OSCE
Q9b What is your experience of the examiner behaviour in the vivas?
Again, generally positive responses were returned (table 5). Interestingly, 58% of respondents felt that the examiners gave feedback (positive or negative) to their responses during the viva. Others found examiner questioning to be closed and examiner timing of the vivas poor such that candidates did not have the chance to cover all available marks.
Examiner behaviour in the vivas
Discussion
The questionnaire response rate of 42% is lower than we had hoped for. Of those who replied 77% passed the FCEM at the first sitting and 95% had completed the FCEM, 65% in the previous 12 months. These data should be borne in mind when interpreting the results since the study population may not be truly representative because the first time pass rate is high and nearly all responded after completing the FCEM. Those who passed pre-2007 are less well represented, perhaps because they may be more likely to have moved on to Consultant posts, changed email address and have less immediate interest in the FCEM than those who recently sat for it.
It is acknowledged that some survey questions may have been ambiguous and led to misinterpretation by candidates. For example in question 2, ‘did the SAQ questions follow what they would ask?’ was meant to ascertain if the SAQ questions covered relevant clinical content but could be construed to ask about the flow of an individual question from stem to end or about question spotting in a more general sense. It is also acknowledged that we did not define response answers such as ‘very surprised’ or ‘slightly different’, though they were used as part of a graded response.2 At the time of writing, the CA section of the exam has changed from a viva examination to a written paper and so some of the answers given for CA preparation in this survey may not be directly applicable to the current format.
The answers to questions 1 and 2 demonstrate that the majority of candidates felt they were ‘well’ to ‘extensively’ prepared for their first attempt at the FCEM. However, in 2006–2007 at least, there was still a sizeable number presenting for examination unclear about what was being tested (content) and how it was being tested (format). Such findings are alarming and may reflect personal preparation, local training issues or wider national specialty training matters.
It is clearly important to be aware exactly what is being tested in all sections of the FCEM and how each is assessed and scored. This is reflected in the answers to questions 3–8 that rate specific exam practice as the most important factor in overall preparation for the FCEM, and for each component individually. Private study is understandably valued, as is small group revision, but it appears that practicing technique under exam timing, understanding marking matrices and familiarisation with the format and style of the FCEM in general are essential for success.
Every trainee should have access to the CEM website1, the guidance and opinion of dedicated EM trainers familiar with all aspects the FCEM and a teaching programme mapped to the curriculum.3 Other resources open to all trainees include revision courses, textbooks, journal articles, guidelines, e-learning modules and revision courses. Some of these have been collated on dedicated websites.1 ,7 ,8
Trainees in smaller Deaneries may find themselves preparing for the FCEM in isolation, and therefore not having the benefits of small group revision such as pulling of resources, exam practice and support. Not all trainees who wanted were able to attend revision courses, though the reasons for this were not consistent. The importance of attending specific FCEM revision courses varied considerably with many finding them invaluable and some choosing to avoid them completely. Of specific component courses, those tackling CA were believed to be most valuable but management courses were often thought to have little relevance for success in the FCEM. Certain regions run yearly mock examinations which have the benefit of ensuring that both the trainees and the trainers are familiar with all aspects of the exam and that surprises for the trainees come well in advance of sitting the FCEM in anger. It requires significant effort to run a formal mock examination but where these are established, they are highly valued as preparation for the FCEM.
Given this, it is difficult to understand how a trainee should get to the point of sitting the FCEM without having a clear idea of what is ahead. The CEM's most recent advice that every ED should have an FCEM examiner may improve this problem.
Assessment of examiner behaviour during the OSCE and viva examinations was generally positive though with fairly regular anecdotal perception of subjectivity, and a feeling that marks were awarded more on examiner impression rather than by the use of a structured template. This was particularly associated with the CTR viva in which candidates regularly suspected that examiners had not read the CTR in sufficient depth. In many such instances, candidates felt that examiner opinion could dominate and precluded adequate assessment of all aspects of the piece of work at interview.
The benefits and limitations of the OSCE are well known and scenarios can often be felt to be unrealistic in terms of both content and timing. Over the last few years, there has been increased transparency of assessment and candidate instructions in the OSCE (such as the pie chart) have improved understanding and are welcomed. Actors now receive more detailed briefing and there is less variability in general.
Conclusions
Exam guidance from the CEM should be improved with particular attention paid to practicalities of sitting each section and suggested revision resources and how to access them. In particular, trainees would value more in the way of specimen exam material being made available. The CEM website1 should be a core source of reference for all parts of the FCEM yet it only appeared in the top five resources in preparation for the CTR, likely due to a great improvement in guidance for this section in 2007.
At the time of survey the CEM curriculum3 was felt to be of limited use but as of 2010 this has been changed to a topic-based curriculum that is more user friendly.9 The CEM initiative of exam information days were also seen as being an important step forward in clarifying what was expected in the FCEM both for trainees and trainers. On a local level, training programme directors should ensure that committed trainers with a good knowledge of the FCEM examination are present in each training department and that each trainee should have access to at least four out of the top five resources for preparation for each component of the FCEM (table 3). Where structured journal clubs and yearly mock examinations are not yet established, careful consideration should be given to doing so.
There will always be isolated criticism regarding a high stakes examination such as the FCEM when so much depends on the outcome. The CEM is required to set standards for the FCEM which are agreed and monitored by the GMC. Regional variations in pass rates and individual examiners performance are also monitored by the CEM. It was generally recognised by those surveyed that the FCEM represented a substantial and fair test. However, every effort should be made to improve this, as with other aspects of training, as the specialty of EM continues to develop in the UK.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Download Supplementary Data (PDF) - Manuscript file of format pdf
Footnotes
-
Competing interests There are no competing interests with regard to any of the authors though Wayne Hamer has served as Chairman of the Training Standards Committee of the College of Emergency Medicine.
-
Ethics approval This was a survey of Emergency Medicine Trainees and was authorised by the Training Standards Committee of the College of Emergency Medicine.
-
Provenance and peer review Not commissioned; externally peer reviewed.
-
↵i When this study was completed the critical appraisal section of the FCEM was a 15 min viva, which has now been replaced by a written module.