Most Common Survey Biases and How To Avoid Them

Among the most common survey biases are sampling bias, order bias, acquiescence bias, social desirability bias, and demand characteristics bias. For each type of bias, a selection of methods to reduce or avoid the bias has been provided.

Table of Contents

Sampling Bias

Sampling bias occurs when the survey sample does not accurately represent the population.
One example of sampling bias is undercoverage, which is when certain members of the population are underrepresented in the sample. This is often a problem of convenience sampling.
Another example of sampling bias is non-response bias. This is when individuals selected for the survey are unwilling or unable to respond to the survey.
A third example of sampling bias is voluntary response bias. When sample members are self-selected, they tend to over-represent individuals with strong opinions.

Avoiding Sampling Bias

In order to avoid sampling bias, it is important to select a sample that represents the population accurately.
One way to do this is to distribute the survey in a way that every type of respondent has access to it. This can include using several channels of distribution.
To avoid sampling bias, it is important to consider what feedback is desired and the customer journey, to ensure these match.
To avoid non-response bias, adjust the survey time to one more relevant to the customer, such as asking for customer service feedback immediately after customer service has been provided.
Creating short surveys, and making sure that respondents know that it will be short, also helps to avoid non-response bias.

Order Bias

The order of the questions can prime respondents into given certain responses, as it appeals to the desire to give internally consistent answers.
Mentioning products or brands in previous questions can also affect a respondent’s response in a later question if asked about familiarity or awareness.
The answer order can also create a bias. There are two types of bias within this: primacy bias, in which a respondent selects one of the first options before reading them all; and recency bias, where the respondent chooses the last option, as this was read most recently and so is more memorable.
In online and print surveys, there is often a bias towards the first few answer options, while with phone and in-person surveys, where all the questions are read out, recency bias is more common.

Avoiding Order Bias

When creating a survey, it is important to consider the order of the questions and to make sure that the previous questions do not influence the responses of following questions.
With a satisfaction survey, general questions should come first, before being followed by specific questions about each part of the experience.
To avoid question order bias, randomize the order of questions within each topic.
For multiple-choice questions where the ordering of the answers doesn’t matter, randomizing the answer order will reduce answer order bias.
Another way to avoid answer order bias is to limit the choice of answers, providing a free response option where necessary.
Creating engaging questions helps to reduce order bias, as it reduces the chance the respondent will just select the first or last answer they see and makes it more likely they will respond honestly.

Acquiescence Bias

Acquiescence bias describes the tendency a respondent has to agree with the questions and statements.
This agreement bias can often result in a bias towards positive responses, without them being a true and honest reflection of the respondents’ thoughts or beliefs.
Respondents are likely to default to a positive response, especially with long or complicated surveys.
Acquiescence bias can often skew answers to an extreme as survey respondents will ignore the more neutral responses in a survey, not believing a “middle-ground” exists, in favor of the stronger responses. This overlaps with other biases that skew towards the extremes.

Avoiding Acquiescence Bias

One way to avoid acquiescence bias is to not pose questions that imply that there is a correct answer. Questions should instead look for a point of view.
Another way to avoid acquiescence bias is to reduce the number of yes-no answers and instead have the respondent choose from alternatives or a ranking.
Leading questions should be avoided, as these will dictate what the survey maker wants to hear and will influence the respondent to choose this answer.
Certain response scales do not have acquiescence bias as strongly, for example: “Definitely will not, Probably will not, Don’t know, Probably will, Definitely will”.

Social Desirability Bias

Desirability bias describes a bias created when respondents answer survey questions in a way to reflect characteristics and behaviors that are socially desirable.
Social desirability often happens when respondents want to create a favorable impression, or do not want to have a negative evaluation.
Desirability bias is often found with sensitive or controversial questions. With certain questions, most respondents will give the socially acceptable response and it is difficult to find truthful answers.

Avoiding Social Desirability Bias

This kind of bias can be reduced by allowing anonymous responses and ensuring the respondent understands the survey is anonymous so that their responses cannot be linked to them.
Distancing the brand behind the survey from the survey itself can also reduce desirability bias, as it reduces the fear of providing an undesirable answer.
Neutral questions should be chosen that avoid social desirability bias.
One way to identify respondents who are strongly affected by social desirability bias is to use a social desirability scale that identifies where respondents fall on a scale, allowing them to be included or excluded in the survey.
By choosing the wording of questions carefully, social desirability bias can be avoided, as the respondent can be assured all the answers are acceptable and that there is no correct response.

Demand Characteristics Bias

Demand characteristics bias describes the bias created when respondents answer in accordance with preset expectations of what the survey wants.
Respondents will be influenced by the fact that they are in a survey and by what they believe the goal of the survey is.
Respondents may want to give responses that please the researcher or that will help the survey achieve its goal, regardless of whether the responses are truthful and accurate to their personal beliefs.
Wording bias can often affect this. For example, if the respondent knows the survey creator, being addressed in an informal way may result in different responses than if it were more formal.

Avoiding Demand Characteristics Bias

To avoid this kind of bias, create engaging and interesting questions that make the respondent forget that they are part of research and instead focus on answering honestly.
A formal approach should be taken with all the survey respondents, whether or not the researcher knows the respondents personally.
Another way of avoiding demand characteristics bias is to disguise the true purpose of the survey. This can be amplified with a double-blind study, where the researchers interacting with the respondents also do not know the purpose of the survey.