“Interviewer” Effects in Face-to-Face Surveys: A Function of Sampling, Measurement Error, or Nonresponse?

Brady T. West, Frauke Kreuter, Ursula JaenichenJournal of Official Statistics 2013 · DOI

Abstract

Recent research has attempted to examine the proportion of interviewer variance that is due to interviewers systematically varying in their success in obtaining cooperation from respondents with varying characteristics (i.e., nonresponse error variance), rather than variance among interviewers in systematic measurement difficulties (i.e., measurement error variance). This study aims to replicate initial work—previously limited to CATI surveys—by analyzing data from a national CAPI survey in Germany where interviewers were responsible for working a fixed subset of cases.

Introduction

In survey methodology, it is well-established that responses collected by the same interviewer tend to exhibit higher similarity than responses collected across different interviewers. Traditionally, this "interviewer effect" has been attributed primarily to measurement error—the idea that an interviewer’s personality, probing style, or demographic characteristics might systematically bias the answers provided by respondents.

However, a second potential source of variance is often overlooked: nonresponse error variance. This occurs when interviewers vary in their ability to recruit specific types of respondents. If one interviewer is particularly effective at engaging younger individuals while another excels with older populations, the resulting datasets will differ even if the measurement process itself is flawless. Distinguishing between these two sources is challenging because researchers typically lack information about non-respondents. This study addresses this gap by utilizing administrative "ground truth" records in a face-to-face (CAPI) survey context.

Methodology: Three-Step Variance Decomposition

The authors employ a multilevel modeling approach to decompose interviewer variance into three distinct components:

  1. Sampling/Assignment Variance: By examining administrative records for all units assigned to an interviewer, the researchers first determine if the initial workloads were balanced. This step accounts for potential geographic clustering or non-random assignment.
  2. Nonresponse Error Variance: The variance in "true" values is then estimated only for the subset of units that agreed to participate. An increase in variance at this stage suggests that interviewers are systematically recruiting different types of people from their assigned pools.
  3. Total Interviewer Variance: Finally, the variance in the actual survey responses is estimated. Comparing this to the nonresponse variance allows the researchers to isolate the noise added during the interview itself (measurement error).

To account for the lack of interpenetrated (random) assignment common in face-to-face surveys, the model includes fixed effects for Primary Sampling Unit (PSU) characteristics, such as local unemployment rates and education levels. This prevents geographic differences from being incorrectly attributed to interviewer behavior.

Data Sources

The study utilizes data from the Labour Market and Social Security (PASS) study in Germany, specifically focusing on the "UB" sample of households receiving unemployment benefits. These survey responses are linked to the Integrated Employment Biographies (IEB), which provide administrative "ground truth" from the German Federal Employment Agency. The final analysis included 4,829 household heads nested within 211 professional interviewers.

Key Findings

The Interviewer as Gatekeeper

The results confirm that interviewers act as significant gatekeepers, with an intra-interviewer correlation for response rates of 0.105. This variability in recruitment success creates a significant opportunity for nonresponse error to influence the final data.

"Who" vs. "How"

The source of interviewer variance depends heavily on the nature of the variable:

  • Objective Variables: For variables like age, the interviewer effect is almost entirely a function of recruitment (nonresponse error).
  • Complex or Sensitive Variables: For variables such as "Unemployment Benefit Receipt," the effect is mixed. Approximately 38% of the variance was attributed to nonresponse, while 62% stemmed from measurement error—likely due to how the question was phrased or perceived.
  • Employment Status: This variable showed a dramatic 700% increase in variance between the assigned sample and reported values, driven almost exclusively by measurement error.

The Role of Geography

A critical insight is the "geography trap." Without controlling for PSU-level covariates, interviewer effects appear significantly larger. In face-to-face surveys, what looks like an interviewer's personal bias is often merely a reflection of the socio-economic characteristics of the neighborhood they were assigned to work in.

Conclusion and Practical Implications

The study demonstrates that for many key variables, interviewer effects are real but their origins vary. For survey practitioners, this suggests that different interventions are needed depending on the goal. If the concern is recruitment bias, interviewer training should focus on standardized contact protocols. If the concern is measurement bias, training should emphasize neutral probing and question delivery.

Furthermore, the authors suggest that field supervisors could use Empirical Best Linear Unbiased Predictors (EBLUPs) to identify interviewers exhibiting extreme recruitment or measurement patterns in real-time, allowing for targeted retraining during the data collection phase. Future research should explore multiple imputation frameworks to extend these findings to datasets where administrative records are not universally available.

Summary updated

Abstract

Abstract Recent research has attempted to examine the proportion of interviewer variance that is due to interviewers systematically varying in their success in obtaining cooperation from respondents with varying characteristics (i.e., nonresponse error variance), rather than variance among interviewers in systematic measurement difficulties (i.e., measurement error variance) - that is, whether correlated responses within interviewers arise due to variance among interviewers in the pools of respondents recruited, or variance in interviewer-specific mean response biases. Unfortunately, work to date has only considered data from a CATI survey, and thus suffers from two limitations: Interviewer effects are commonly much smaller in CATI surveys, and, more importantly, sample units are often contacted by several CATI interviewers before a final outcome (response or final refusal) is achieved. The latter introduces difficulties in assigning nonrespondents to interviewers, and thus interviewer variance components are only estimable under strong assumptions. This study aims to replicate this initial work, analyzing data from a national CAPI survey in Germany where CAPI interviewers were responsible for working a fixed subset of cases.
BibTeX
@article{west2013interviewer,
  author = {Brady T. West and Frauke Kreuter and Ursula Jaenichen},
  title = {{“Interviewer” Effects in Face-to-Face Surveys: A Function of Sampling, Measurement Error, or Nonresponse?}},
  journal = {Journal of Official Statistics},
  year = {2013},
  doi = {10.2478/jos-2013-0023},
}
Figure