Methods

The Journal of Accident and Emergency Medicine (JAEM) was hand searched from 1994 to 1998 inclusively (the journal was first published under this title in 1994). All questionnaire based studies of A&E departments were identified and reviewed by a single person (PB) to assess: how departments were selected for inclusion; the number of departments studied; the study design and response rates. Data were entered and analysed on an Excel spreadsheet. Calculation of the Pearson correlation coefficient (r) using SPSS,³ assessed the linear association between sample size and response rates.

Results

Thirty questionnaire based studies were published in the JAEM during the study period. Twenty two were postal surveys, six telephone surveys, one a postal survey with telephone follow up and in two cases the method was not stated. Only two topics were the subject of more than one questionnaire; these subjects were violence in A&E and the role of nurse practitioners. With regard to violence in A&E, one study looked at 114 A&E departments in five regions and the other looked at 310 departments in the UK and Ireland. One study of nurse practitioners looked at all departments (n=465) and one looked at major departments (n= 293).

Three (10%) of the studies did not quote a response rate and one could not be calculated. Twenty three studies reported a single response rate; three studies reported two response rates and one study had three response rates (because of questioning two professional groups, splitting major and minor departments or using two questionnaires at different points in time).

The sample sizes of the identified studies ranged from 15 to 742 with a mean of 178.5 (standard deviation: 143.75) and a median of 136. No sample size was stated for one study. Response rates ranged from 55% to 100% with a mean of 79% and a standard deviation of 14.6. In eight (26%) of the 31 studies where a response rate was available this was less than 70%. Higher response rates were associated with decreasing sample size (fig 1) (Pearson's r= −0.389, p<0.05).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Association between sample size and response rate.

Twenty one (67.7%) studies were structured to receive one reply from each department. However, inclusion criteria were variable. The geographical area studied was a single region in one case, four regions in one case, England in two cases, England and Wales in three cases, the United Kingdom in five cases, the United Kingdom and Ireland in five cases and not stated in three cases. The size of department defined by attenders was variable with one study each for a lower limit of 10, 20, 30, 35 and 40 thousand attenders and one stating “major” department without definition. Fourteen surveys did not place a limit on attendances and three restricted their study to consultant led departments.

Nine studies requested opinion from individual people; they are summarised in table 1.

Seventeen (54.8%) studies did not state the source of their information about departments or people surveyed, nine used the BAEM handbook, one the Directory of Emergency Care and Special Units, one both of these, one used a BAEM mailing list and two used local contacts. A further two used local databases.

Two studies differentiated results on the size of the department and one on location (rural/urban). No study differentiated results according to whether the department was consultant led.

Discussion

The limitations of this study are that it only reports on those questionnaire studies reported in one journal. The sample is small but we believe the problems highlighted need to be drawn to the attention of potential researchers at an early stage to prevent these problems continuing in the future.

Questionnaire studies relating to A&E do not use a consistent well defined population of A&E departments. Information in the studies is usually inadequate to allow them to be repeated.

The methods sections of many of these questionnaire based studies are incomplete. They fail to detail how and why the sample was selected. This means that many studies are not reproducible. The variety of opinion on who to include and the failure to analyse major subsets also means that many studies are not generalisable. Response rates and sample sizes are highly variable and larger studies seem to have lower response rates. However, the strength of this association is dependent on the inclusion of one large study (n=742) that had a low response rate (55%). Over a quarter (26%) of studies had response rates of less than 70% and the reliability of data based on surveys with low return rates must be questioned.

We believe all questionnaires should include limitations sections, as recently suggested,⁴ so that factors limiting generalisability are explicit. Conclusions based on majority decisions should not be presumed to be correct for all. A specialist is a person who specially or exclusively studies a subject⁵ and therefore such persons should be expected to have a better evidence base to their practice of medicine. Consultant run departments should therefore be analysed separately.

Results should be analysed according to the nature of the department (for example, consultant led, number of new attenders, urban/rural) so that readers can extrapolate the results to their own working environment.