Nonresponse and Measurement Error Variance among Interviewers in Standardized and Conversational Interviewing

Brady T. West, Frederick G. Conrad, Frauke Kreuter, Felicitas MitterederJournal of Survey Statistics and Methodology 2017 · DOI

Abstract

Recent methodological studies have attempted to decompose the interviewer variance introduced in interviewer-administered surveys into its potential sources using the Total Survey Error framework. This study examines the relative contributions of nonresponse error variance and measurement error variance to total interviewer variance across two techniques: Standardized Interviewing (SI) and Conversational Interviewing (CI). Using a randomized experiment in Germany linked to administrative records, we find that measurement error variance is generally the more significant source of total interviewer variance, regardless of the interviewing technique employed. These findings have important implications for how survey practitioners train interviewers and design data collection protocols.

Introduction

In the field of survey methodology, interviewer variance—the phenomenon where different interviewers produce systematically different results from similar respondent pools—remains a significant challenge. This variance increases the standard errors of survey estimates, effectively reducing the precision of the data and shrinking the "effective" sample size.

Traditionally, this "cluster effect" has been attributed to the interviewer's behavior during the question-and-answer process (measurement error). However, interviewers also act as recruiters. If one interviewer is particularly effective at convincing young people to participate while another excels with the elderly, they introduce nonresponse error variance. This paper addresses a major gap in the literature: does the interviewing technique—specifically the rigid Standardized Interviewing (SI) versus the flexible Conversational Interviewing (CI)—alter the balance between these two error sources?

Methodology

The study utilized an interpenetrated sampling design within a large-scale experiment in Germany. A national sample of 7,200 employed individuals was selected from the Integrated Employment Biographies (IEB) database, a high-quality administrative record system. These cases were randomly assigned to 60 professional interviewers, who were themselves randomized into two groups:

  1. Standardized Interviewing (SI): Interviewers were required to read questions exactly as scripted and provide only neutral, non-directive probes (e.g., "Let me repeat the question").
  2. Conversational Interviewing (CI): Interviewers were encouraged to use their own words to clarify definitions and ensure the respondent understood the underlying intent of the questions.

By comparing survey responses to the "ground truth" administrative data for variables like birth year, income, and unemployment duration, the researchers employed a four-step multilevel modeling approach to isolate variance stemming from sampling, unit nonresponse, item nonresponse, and measurement error. The final analysis was based on 1,850 completed face-to-face interviews.

Key Findings

The Dominance of Measurement Error

The primary finding of the study is that measurement error is the dominant contributor to total interviewer variance for most complex survey items. For variables such as annual income and employment history, the "noise" introduced by interviewers occurred during the data collection phase rather than the recruitment phase. For example, in the SI group, the Intraclass Correlation (IIC) for income was 0.021, almost entirely attributed to measurement error.

The "Cancelling Out" Effect

One of the most intriguing observations involved reports of employment duration. In the CI group, researchers identified significant variance at the recruitment stage (nonresponse error). However, this variance largely disappeared in the final reports. This suggests a "cancelling out" effect, where the measurement process effectively offset the initial recruitment bias, pulling disparate respondent groups toward a more uniform mean. While this reduces variance, it does not necessarily imply higher accuracy, but rather a homogenization of the data by the interviewer.

Technique and Variance

A common concern among methodologists is that the flexibility of Conversational Interviewing might lead to higher interviewer-level variance because interviewers are "off-script." However, the results indicate that CI does not consistently lead to higher measurement variance than the rigid SI method. In fact, while CI is often superior for accuracy, this study suggests it does not come at the cost of significantly higher interviewer-level noise.

Simple Facts and the Gatekeeper Effect

The source of variance appears to depend heavily on the nature of the question. For simple, factual questions like "birth year," the variance was almost entirely driven by nonresponse error (IIC = 0.025 in the CI group). This indicates a "gatekeeper effect," where certain interviewers are more effective at recruiting specific age cohorts, but once the interview begins, there is little room for measurement error to manifest in such a straightforward variable.

Conclusion and Implications

This research highlights that while interviewers influence data as both recruiters and collectors, their role as data collectors remains the primary source of variance in complex measures. Importantly, this holds true regardless of whether interviewers follow a rigid script or engage in flexible, conversational clarification.

For survey practitioners, these findings suggest that efforts to reduce interviewer variance should continue to prioritize the measurement process and interviewer training regarding question delivery. Furthermore, the lack of increased variance in CI supports its viability as a technique that can improve data quality without necessarily introducing excessive interviewer-level instability. Future research should explore the "liking" hypothesis—whether demographic similarity between interviewers and respondents affects these error sources—and replicate these findings in different cultural contexts.

Summary updated

Abstract

Recent methodological studies have attempted to decompose the interviewer variance introduced in interviewer-administered surveys into its potential sources, using the Total Survey Error framework. These studies have informed the literature on interviewer effects by acknowledging interviewers’ dual roles as recruiters and data collectors, thus examining the relative contributions of nonresponse error variance and measurement error variance among interviewers to total interviewer variance. However, this breakdown may depend on the interviewing technique: some techniques emphasize behaviors designed to reduce variation in the answers collected by interviewers more so than other techniques. The question of whether the contributions of these error sources to total interviewer variance change for different interviewing techniques remains unanswered. Addressing this gap in knowledge has important implications for interviewing practice because the technique used could alter the relative contributions of variance in these error sources to total interviewer variance. This article presents results from an experimental study mounted in Germany that was designed to answer this question about two specific interviewing techniques. A national sample of employed individuals was first selected from a database of official administrative records, then randomly assigned to interviewers who themselves were randomized to conduct either conversational interviewing (CI) or standardized interviewing (SI), and finally measured face-to-face on a variety of cognitively challenging survey questions with official values also available for verifying the accuracy of responses. We find that although nonresponse error variance does exist among interviewers for selected measures (especially respondent age in the CI group), measurement error variance tends to be the more important source of total interviewer variance, regardless of whether interviewers are using CI or SI.
BibTeX
@article{west2017nonresponse,
  author = {Brady T. West and Frederick G. Conrad and Frauke Kreuter and Felicitas Mittereder},
  title = {{Nonresponse and Measurement Error Variance among Interviewers in Standardized and Conversational Interviewing}},
  journal = {Journal of Survey Statistics and Methodology},
  year = {2017},
  doi = {10.1093/jssam/smx029},
}
Figure