Does a low survey response rate mean the results are biased?

By ·

Short answer: Not automatically. A large meta-analysis found that response rate alone is a weak predictor of nonresponse bias [S1]. Bias grows when the reasons people don't respond are linked to what you measure. In public services, access to the survey channel is often tied to the experience being studied, so a response rate can hide the gap [S2][S3].

1. Response rate alone doesn't tell you whether results are biased

It is natural to read a low response rate as a warning sign and a high one as reassurance. The research evidence is more specific than that.

Robert Groves and Emilia Peytcheva pooled 59 methodological studies that had measured nonresponse bias directly, by comparing respondents with information known about the full sample [S1]. Two findings matter for anyone commissioning or reviewing a survey:

  • The response rate explained little of the bias. Surveys with similar response rates showed very different levels of bias.
  • Bias varied more across questions within the same survey than between surveys. A single study could be close to unbiased on one measure and noticeably biased on another.

So "our response rate was 60%" and "our response rate was 4%" are both incomplete statements about quality. What you need to know is which answers might be distorted, and why.

This doesn't make response rates irrelevant. A low rate leaves more room for bias and makes subgroup estimates less stable. But it is the mechanism behind non-response, not the rate itself, that decides whether a given finding can be trusted.

2. Bias appears when who answers is tied to what you ask about

Groves set out the mechanism plainly in an earlier paper: nonresponse bias for a given estimate depends on how strongly the likelihood of responding is related to the variable being measured [S6]. If people are equally likely to respond regardless of their answer, even a low response rate can give an unbiased estimate. If the people least likely to respond are also the people with a particular experience, the estimate drifts, whatever the response rate.

Public services offer clear examples of this link:

  • Asking about access barriers through a channel that requires access. An online-only survey about whether people could reach a service will under-represent people who couldn't get online.
  • Asking about stressful processes after the fact. People who had the hardest time may be the least willing or able to revisit it.
  • Asking about trust through official channels. People with low trust in an institution may be less inclined to answer that institution's survey.

In each case, the question and the reason for not answering point the same way. That is the situation the meta-analysis flags as high risk [S1].

A practical test for any survey plan: for each key question, ask "Is there a plausible reason why the people with one answer would be less likely to reach, open or finish this survey?" If yes, the response rate tells you very little about that question.

3. Online-only collection misses people unevenly

Digital channels are efficient, and most people are online. But "most" is not evenly distributed, and the gaps line up with groups that public services often most need to hear from.

In the United States, Pew Research Center's 2023 survey (published January 2024) found [S2]:

  • 95% of adults in households earning $100,000 or more a year have home broadband, compared with 57% of adults in households earning under $30,000.
  • 15% of adults are "smartphone dependent": they have a smartphone but no home broadband. Among adults in households earning under $30,000, the figure is 28%.

Smartphone-dependent respondents can often complete a short survey, but long forms, file uploads or open-ended writing on a small screen are harder. A survey that "works on mobile" may still lose people partway through.

In Canada, Statistics Canada's Canadian Internet Use Survey reports that 95% of people aged 15 and older used the internet in 2022, while among people aged 75 and older the figure was 72% [S3]. An Employment and Social Development Canada research summary based on the same survey estimates that about 1.8 million Canadians didn't use the internet at all in 2022; most were seniors, not working or living alone, and many were women or lived in rural areas [S4]. The same summary notes that people with lower income and education, and people with disabilities, were more likely to be minimal users rather than intensive ones [S4].

None of this means online surveys are wrong. It means an online-only design builds in a predictable coverage gap, and that gap is larger for some populations than others.

4. Multiple channels are a design decision, not an add-on

If the risk comes from a link between access and experience, the remedy has to be built into the design rather than patched on after a disappointing response.

The OECD's Guidelines for Citizen Participation Processes treat recruitment and the choice of participation methods as deliberate planning steps, made with the intended participants in mind, rather than defaults [S5]. Applied to surveys and story collection, that means deciding at the start:

  • Who must be able to take part for the findings to be credible, and which of them are least likely to reach the default channel.
  • Which channels reach them: phone, paper, in-person collection, assisted completion by trusted intermediaries, or a combination.
  • How each channel will be recorded, so analysts can compare results by channel and readers can judge coverage.

Designing Sensemaking Studies puts the principle this way: "In sensemaking studies, broad participation is a data-quality requirement: if a meaningful part of the population can't participate, findings can reflect access more than experience." [BOOK, p.4]

Two cautions. First, adding channels can change how people answer (a "mode effect"), so record the channel and check for differences rather than pooling blindly. Second, an extra channel only helps if it is actually offered to the people who need it. A paper option that must be requested online doesn't solve the problem.

5. What one tribunal did

The BC Employment and Assistance Appeal Tribunal hears appeals about income and disability assistance decisions. Its mailed feedback surveys came back at a rate of one or two a year, and roughly 30% of appellants lacked computer or phone access. Rather than relying on a single channel, law-student volunteers collected experiences by phone and on paper [CASE].

The lesson isn't the specific tools. It is that the tribunal treated reachability as part of the research question: the people hardest to reach were among the people whose experience mattered most.

6. A checklist for your next collection plan

Before fieldwork, write down answers to these questions:

  1. Who might be missing? List the groups least likely to reach your default channel, and why (connectivity, disability, language, literacy, trust, time).
  2. Is their absence linked to what you're asking? For each key question, note whether non-response could plausibly be tied to the answer.
  3. Which channels reach them? Name the channels and who will run them (staff, volunteers, partner organisations).
  4. How will you record the channel? Capture it with each response so you can compare and report it.
  5. What will you compare against? Use what you already know about the population (administrative records, caseload data, census figures) to check who responded against who you meant to reach.
  6. How will you report who may still be missing? State the remaining gaps in the findings, rather than leaving readers to assume full coverage.

A response rate belongs in the report. It just shouldn't be the only evidence offered for whether the findings hold.

Sources