Nonresponse Bias and the Limits of Paradata

This direction is merged into another direction: response rate bias. Its last version stays readable and citable here.

Version 1 of 14 papers17 conceptsfrozen 2026-10-04

For decades, the "response rate" was the gold standard for survey quality. The logic was simple: if 90% of people answer your survey, the results are likely accurate; if only 20% answer, the results are suspect. However, as participation rates have plummeted globally, researchers have discovered that this intuition is often wrong.

In Issues Facing the Field: Alternative Practical Measures of Representativeness of Survey Respondent Pools, Robert M. Groves and colleagues argue that response rates are poor predictors of nonresponse bias. A low response rate only creates the potential for bias; actual bias only occurs if the reason people don't respond is directly related to the topic you are measuring.

The Mechanism of Bias

To understand why a survey might be biased, we have to look at the relationship between three things: the likelihood of participating, the survey's target variable (e.g., health status), and "auxiliary variables" (data we have for everyone, even non-respondents).

As Aaron Maitland and Carolina Casas-Cordero explain in An Evaluation of Nonresponse Bias Using Paradata from a Health Survey, nonresponse bias is a function of the correlation between participation and the outcome. If healthy people are more likely to answer a health survey than sick people, the results will overstate the population's health.

To "fix" this statistically, researchers use auxiliary variables to weight the data. However, for a variable to effectively correct bias, it must meet a strict dual-correlation requirement:

  1. It must correlate with the probability of participation.
  2. It must correlate with the survey outcome itself.

If a variable only predicts who answers the door but has nothing to do with the survey topic, using it for statistical adjustment will fail to reduce bias.

The Promise and Failure of Paradata

In recent years, researchers have turned to "paradata"—the administrative data generated during the collection process—as a potential savior. Paradata includes things like the number of call attempts, whether a respondent said they were "too busy," or if an interviewer noted a locked gate at the residence.

The appeal of paradata is that it is available for both respondents and non-respondents, making it a candidate for the auxiliary variable "bridge." But as Frauke Kreuter notes in Paradata - big data of survey research, while this data is excellent for monitoring operational efficiency, it often fails to correct bias.

The problem is that most paradata is "topic-blind." Knowing that a respondent was "too busy" to talk tells you a lot about why they didn't participate (fulfilling the first requirement), but "busyness" rarely correlates strongly with specific outcomes like chronic disease or political preference (failing the second requirement). In An Exploration into the Use of Paradata for Nonresponse Adjustment in a Health Survey, Maitland and Casas Cordero demonstrate that general process metrics like interviewer effort show negligible correlation with health outcomes, rendering them weak tools for adjustment.

Moving Toward Topic-Specific Proxies

The path forward involves shifting from general process metadata to domain-specific proxy indicators. Instead of just recording how we contacted someone, researchers are beginning to record what we can observe about them that relates to the study.

For a health survey, this might mean training interviewers to look for physical cues—such as whether a home has a wheelchair ramp or if the respondent mentions being ill during the initial doorstep interaction. By capturing these "topic-relevant" observations during the recruitment phase, researchers create the necessary link between participation behavior and the variables of interest. This allows for a much more precise statistical correction, moving the field beyond the blunt instrument of response rates toward a more nuanced understanding of representativeness.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_nonresponse_bias_and_the_limits_of_parad_30dc844f,
  title = {Nonresponse Bias and the Limits of Paradata},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/nonresponse-bias-and-the-limits-of-paradata/rdr_30dc844fa38b4dc88972f0a2289225c1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_nonresponse_bias_and_the_limits_of_parad_30dc844f_v1,
  title = {Nonresponse Bias and the Limits of Paradata},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/nonresponse-bias-and-the-limits-of-paradata/rdr_30dc844fa38b4dc88972f0a2289225c1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}