Background—Questionnaires are commonly sent to accident and emergency (A&E) departments to determine common practice and are often extrapolated to best practice.
Aims—To determine if questionnaire based studies have a defined population of A&E departments and whether studies are reproducible.
Methods—All questionnaires in the Journal of Accident and Emergency Medicine were reviewed and assessed for inclusion criteria, departments studied and study design.
Results—30 questionnaires were detected, 22 were postal, six telephone and two did not state method of contact. Sample sizes ranged from 15 to 740 and inclusion of A&E departments was highly variable according to geographical area, size of department or consultant status. Seventeen (54.8%) did not state the source of A&E department listings. Response rates ranged from 55–100%. Only three studies undertook subset analysis according to either size or locality.
Conclusions—Questionnaire studies of A&E departments have poor methodology descriptions, which means that many are not reproducible. Inclusion criteria are highly variable and failure to analyse important subsets may mean that individual departments cannot apply recommendations. Questionnaire studies relating to A&E do not use a consistent well defined population of A&E departments. Information in the studies is usually inadequate to allow them to be repeated.
Statistics from Altmetric.com
Questionnaire based studies can be a useful means of determining perceptions of current or best practice and methods of constructing them have been described elsewhere by the authors.1 Whether the majority opinion or decision should change a department's or individual's practice will depend on the generalisability and reliability of the study. Reliability is used to describe whether a questionnaire can be trusted to give consistent results.2 The term generalisability relates to whether the findings of a particular study are applicable to other populations2 and is dependent on both the quality of the questionnaire and whether the study population is representative of the target population.
The criterion used in the selection of the departments to be studied is an important issue when assessing generalisability. Departments that are exemplars of good practice are the ideal sources of information but given the difficulties in identifying this population a proxy measure has to be used when conducting a survey. The sample selected may be based on the size of the department, whether the department is run by specialists or other criteria. Accident and emergency (A&E) departments led by specialists in A&E medicine should represent best practice. Larger departments may have more resources and educational programmes and so represent leading edge departments. Smaller units may, however, have greater flexibility. Studies of large departments may not be applicable to the smaller units or minor injury units and studies of rural departments may not be applicable to urban areas. It is therefore vital that the methods section of any research publication defines the exact characteristics of population surveyed and that they are taken into account when deciding whether the results of an analysis are relevant to any individual department.
The aim of this study was to determine if questionnaire based studies relating to A&E departments have a consistent defined population of A&E departments.
The Journal of Accident and Emergency Medicine (JAEM) was hand searched from 1994 to 1998 inclusively (the journal was first published under this title in 1994). All questionnaire based studies of A&E departments were identified and reviewed by a single person (PB) to assess: how departments were selected for inclusion; the number of departments studied; the study design and response rates. Data were entered and analysed on an Excel spreadsheet. Calculation of the Pearson correlation coefficient (r) using SPSS,3 assessed the linear association between sample size and response rates.
Thirty questionnaire based studies were published in the JAEM during the study period. Twenty two were postal surveys, six telephone surveys, one a postal survey with telephone follow up and in two cases the method was not stated. Only two topics were the subject of more than one questionnaire; these subjects were violence in A&E and the role of nurse practitioners. With regard to violence in A&E, one study looked at 114 A&E departments in five regions and the other looked at 310 departments in the UK and Ireland. One study of nurse practitioners looked at all departments (n=465) and one looked at major departments (n= 293).
Three (10%) of the studies did not quote a response rate and one could not be calculated. Twenty three studies reported a single response rate; three studies reported two response rates and one study had three response rates (because of questioning two professional groups, splitting major and minor departments or using two questionnaires at different points in time).
The sample sizes of the identified studies ranged from 15 to 742 with a mean of 178.5 (standard deviation: 143.75) and a median of 136. No sample size was stated for one study. Response rates ranged from 55% to 100% with a mean of 79% and a standard deviation of 14.6. In eight (26%) of the 31 studies where a response rate was available this was less than 70%. Higher response rates were associated with decreasing sample size (fig 1) (Pearson's r= −0.389, p<0.05).
Twenty one (67.7%) studies were structured to receive one reply from each department. However, inclusion criteria were variable. The geographical area studied was a single region in one case, four regions in one case, England in two cases, England and Wales in three cases, the United Kingdom in five cases, the United Kingdom and Ireland in five cases and not stated in three cases. The size of department defined by attenders was variable with one study each for a lower limit of 10, 20, 30, 35 and 40 thousand attenders and one stating “major” department without definition. Fourteen surveys did not place a limit on attendances and three restricted their study to consultant led departments.
Nine studies requested opinion from individual people; they are summarised in table 1.
Seventeen (54.8%) studies did not state the source of their information about departments or people surveyed, nine used the BAEM handbook, one the Directory of Emergency Care and Special Units, one both of these, one used a BAEM mailing list and two used local contacts. A further two used local databases.
Two studies differentiated results on the size of the department and one on location (rural/urban). No study differentiated results according to whether the department was consultant led.
The limitations of this study are that it only reports on those questionnaire studies reported in one journal. The sample is small but we believe the problems highlighted need to be drawn to the attention of potential researchers at an early stage to prevent these problems continuing in the future.
Questionnaire studies relating to A&E do not use a consistent well defined population of A&E departments. Information in the studies is usually inadequate to allow them to be repeated.
The methods sections of many of these questionnaire based studies are incomplete. They fail to detail how and why the sample was selected. This means that many studies are not reproducible. The variety of opinion on who to include and the failure to analyse major subsets also means that many studies are not generalisable. Response rates and sample sizes are highly variable and larger studies seem to have lower response rates. However, the strength of this association is dependent on the inclusion of one large study (n=742) that had a low response rate (55%). Over a quarter (26%) of studies had response rates of less than 70% and the reliability of data based on surveys with low return rates must be questioned.
We believe all questionnaires should include limitations sections, as recently suggested,4 so that factors limiting generalisability are explicit. Conclusions based on majority decisions should not be presumed to be correct for all. A specialist is a person who specially or exclusively studies a subject5 and therefore such persons should be expected to have a better evidence base to their practice of medicine. Consultant run departments should therefore be analysed separately.
Results should be analysed according to the nature of the department (for example, consultant led, number of new attenders, urban/rural) so that readers can extrapolate the results to their own working environment.
MWC and SW designed the study; PB undertook the review and all the hand searching of studies and created the database. MWC and SW analysed the data. All authors were involved in preparation of the paper. MWC is guarantor of the paper.
Conflicts of interest: none.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.