Abstract
Nonresponse rates are poor indicators of nonresponse error in survey estimates. Despite this, the field and the public continue to use response rates as the primary indicator of survey quality. Aside from sample size, the response rate is arguably the most commonly cited criterion for data integrity. This paper addresses the lack of alternative indicators by proposing a practical framework for measuring representativeness and nonresponse error.
Introduction: The Response Rate Myth
For decades, the "response rate"—the percentage of a sampled population that completes a survey—has served as the gold standard for data quality. The logic was intuitive: the more people who answer, the more representative the data must be. However, modern survey science has increasingly debunked this assumption. Research demonstrates that a survey with a 20% response rate can, in many instances, yield more accurate estimates than one with a 60% response rate.
The critical factor is not the quantity of nonrespondents, but the nature of their absence. If those who decline to participate differ systematically from those who do regarding the survey's key topics, nonresponse bias occurs. Despite this understanding, the field has struggled to move past the response rate because no standardized alternative has gained traction. This paper proposes a new taxonomy of metrics designed to provide a more nuanced and accurate assessment of how representative a respondent pool truly is.
A Hierarchical Framework for Representativeness
The authors argue that nonresponse error is not a monolithic property of a survey. Instead, they propose a three-tiered approach to measuring quality, moving from broad survey-level metrics to highly specific variable-level indicators.
1. Survey-Level Indicators (Single Indicators)
These metrics aim to provide a single number to characterize the entire respondent pool, acting as a direct replacement for the traditional response rate.
- Variance of Weights: This measures the extent to which researchers must "weight" or adjust certain respondents' data to align with known population totals. High variance suggests that the respondent pool is heavily skewed, requiring extreme adjustments that can increase the margin of error.
- R-indexes (Representativeness Indexes): These are model-based measures that utilize auxiliary data to determine how "balanced" the respondent pool is compared to the original sample. A high R-index indicates that the respondents are a good "miniature" of the total sample across known characteristics.
2. Variable-Level Indicators (Individual Estimates)
Perhaps the most significant insight of the paper is that nonresponse error is often specific to a particular variable rather than the survey as a whole. For example, a survey might yield an unbiased estimate for "political affiliation" but a highly biased estimate for "average household income" if high-income individuals are less likely to respond.
- Weight-Variable Correlations: This involves checking if the adjustment weights are highly correlated with specific survey answers. If they are, it serves as a "red flag" that nonresponse is likely biasing that specific estimate.
- Fraction of Missing Information (FMI): Using imputation models, researchers can estimate how much uncertainty is introduced by missing data for a specific question.
3. Set-Level Indicators
As a middle ground, the authors suggest grouping variables into sets (e.g., all health-related questions or all demographic questions) that share similar error profiles. This allows for a more granular view than a single survey-level metric without the overwhelming complexity of analyzing every single variable.
The Necessity of Auxiliary Data
A recurring theme in the proposed framework is the reliance on "auxiliary variables." These are pieces of information known for both respondents and non-respondents, typically drawn from a sampling frame, census data, or administrative records. This auxiliary data acts as a "bridge," allowing researchers to see into the "black box" of nonresponse. Without this information, measuring bias becomes nearly impossible, as there is no baseline for comparison.
Challenges and the Path Forward
While the proposed metrics offer a more rigorous way to evaluate survey quality, the authors acknowledge several hurdles to widespread adoption:
- The Complexity Barrier: Moving from a single percentage (response rate) to a suite of statistical indicators represents a significant leap in complexity for practitioners and the public.
- Data Availability: Many surveys, particularly "cold-call" or opt-in web panels, lack the high-quality auxiliary data required for R-indexes or sophisticated weighting.
- Validation: The paper calls for these indicators to be tested across various ongoing surveys to determine which serve as the most reliable "smoke detectors" for actual bias.
Conclusion
The field of survey research is at a crossroads. While the response rate is a comfortable and familiar metric, its failure as a predictor of nonresponse error is well-documented. By adopting a more nuanced taxonomy of representativeness—one that recognizes that error varies by variable and requires auxiliary data to uncover—researchers can provide a more honest and accurate account of data quality. The goal is to move the conversation from "How many people answered?" to "How much does the difference between respondents and non-respondents matter for this specific result?"
Summary updated