Questionnaire surveys: sources of bias

Answers

The questionnaire survey would have been prone to non-response bias (b), response bias (c), selection bias (d), and volunteer bias (e), but not attrition bias (a).

Non-response bias would have occurred if some people did not respond to the survey and they differed from the responders in their sociodemography, behaviour, or attitudes. The above survey was liable to non-response bias because not all of the sample members who were eligible to participate responded, there being a response rate of 53.4% (b is true). It is of interest why some of the original sample did not respond. However, any differences between the non-responders and responders may be difficult to quantify because limited information, if any, was available for those who did not participate. It would have been unethical to use previously collected information in patient databases, for example, about the non-responders, including their demographics, because they have not consented to be part of the study in any capacity. If non-response bias existed it would have threatened the external validity of the survey—that is, the extent to which the survey results could be generalised to the population.

Non-response bias should not be confused with response bias. Response bias would have occurred if there was systematic distortion in the way respondents answered questions. The survey was prone to response bias (c is true); it was possible the sample members responded about the use of placebo treatments in medical care in a way they perceived was of interest to the researchers. Response bias is a particular problem in questionnaire surveys that investigate socially unacceptable or embarrassing behaviours, such as alcohol consumption or drug taking.

Response bias is one of a group of biases collectively known as ascertainment bias, sometimes referred to as detection bias. Ascertainment bias is the systematic distortion of the assessment of outcome measures by the investigators or study participants. This group of biases is a particular problem in clinical trials when the researchers or trial participants are aware of the treatment allocation.2

Volunteer bias would have occurred if there was a systematic difference between those residents who volunteered to be part of the study (completed the survey) and the population. Although a random sample of residents from Northern California was originally selected, not all of them volunteered. The volunteers would be expected to differ from the original sample, and therefore the population, in their sociodemography, behaviour, and attitudes. Therefore, the above survey would have been prone to volunteer bias (e is true). However, it may be difficult to quantify the differences between the volunteers and the population. If volunteer bias existed it would have threatened the external validity of the survey—that is, the extent to which the survey results can be generalised to the population.

Non-response bias and volunteer bias are often confused. Non-response bias considers any differences between the non-responders and responders originally selected for the sample, whereas volunteer bias focuses on the differences between the volunteers and the population. Both will lead to a similar bias; in particular, the potential for the data collected not to be representative of the population but only those who choose to participate. It has been reported that, in general, those who participate in studies are more educated, come from a higher social class, and are more sociable than those who do not participate.

Selection bias would have occurred if there had been a systematic difference between those patients selected for the survey and those who were not, resulting in a sample that was not representative of the population. The original sample of 1800 adults was not a random sample from the theoretical infinite population but a sample of the eligible residents in northern California. The extent to which the residents of northern California were representative of the patient population was not clear. Therefore, because not all members of the population were eligible for selection, the above questionnaire was prone to selection bias (d is true). However, if the residents of northern California were representative of the population, because a random sample of these residents was selected, the original sample would not be prone to selection bias and d would be false.

Attrition bias occurs in longitudinal studies, such as cohort studies and clinical trials, when people are lost to follow-up in a non-random manner. Bias occurs if those lost to follow-up differ in some systematic way from those not lost to follow-up. It is a problem if the characteristics of those people who are lost to follow-up or the reasons for attrition are associated with the outcome measure(s). The questionnaire survey was a cross sectional study, with respondents surveyed at one point in time, so attrition bias did not occur (a is false).