Abstract
Recent methodological studies have attempted to decompose the interviewer variance introduced in interviewer-administered surveys into its potential sources using the Total Survey Error framework. This study examines the relative contributions of nonresponse error variance and measurement error variance to total interviewer variance across two techniques: Standardized Interviewing (SI) and Conversational Interviewing (CI). Using a randomized experiment in Germany linked to administrative records, we find that measurement error variance is generally the more significant source of total interviewer variance, regardless of the interviewing technique employed. These findings have important implications for how survey practitioners train interviewers and design data collection protocols.
Introduction
In the field of survey methodology, interviewer variance—the phenomenon where different interviewers produce systematically different results from similar respondent pools—remains a significant challenge. This variance increases the standard errors of survey estimates, effectively reducing the precision of the data and shrinking the "effective" sample size.
Traditionally, this "cluster effect" has been attributed to the interviewer's behavior during the question-and-answer process (measurement error). However, interviewers also act as recruiters. If one interviewer is particularly effective at convincing young people to participate while another excels with the elderly, they introduce nonresponse error variance. This paper addresses a major gap in the literature: does the interviewing technique—specifically the rigid Standardized Interviewing (SI) versus the flexible Conversational Interviewing (CI)—alter the balance between these two error sources?
Methodology
The study utilized an interpenetrated sampling design within a large-scale experiment in Germany. A national sample of 7,200 employed individuals was selected from the Integrated Employment Biographies (IEB) database, a high-quality administrative record system. These cases were randomly assigned to 60 professional interviewers, who were themselves randomized into two groups:
- Standardized Interviewing (SI): Interviewers were required to read questions exactly as scripted and provide only neutral, non-directive probes (e.g., "Let me repeat the question").
- Conversational Interviewing (CI): Interviewers were encouraged to use their own words to clarify definitions and ensure the respondent understood the underlying intent of the questions.
By comparing survey responses to the "ground truth" administrative data for variables like birth year, income, and unemployment duration, the researchers employed a four-step multilevel modeling approach to isolate variance stemming from sampling, unit nonresponse, item nonresponse, and measurement error. The final analysis was based on 1,850 completed face-to-face interviews.
Key Findings
The Dominance of Measurement Error
The primary finding of the study is that measurement error is the dominant contributor to total interviewer variance for most complex survey items. For variables such as annual income and employment history, the "noise" introduced by interviewers occurred during the data collection phase rather than the recruitment phase. For example, in the SI group, the Intraclass Correlation (IIC) for income was 0.021, almost entirely attributed to measurement error.
The "Cancelling Out" Effect
One of the most intriguing observations involved reports of employment duration. In the CI group, researchers identified significant variance at the recruitment stage (nonresponse error). However, this variance largely disappeared in the final reports. This suggests a "cancelling out" effect, where the measurement process effectively offset the initial recruitment bias, pulling disparate respondent groups toward a more uniform mean. While this reduces variance, it does not necessarily imply higher accuracy, but rather a homogenization of the data by the interviewer.
Technique and Variance
A common concern among methodologists is that the flexibility of Conversational Interviewing might lead to higher interviewer-level variance because interviewers are "off-script." However, the results indicate that CI does not consistently lead to higher measurement variance than the rigid SI method. In fact, while CI is often superior for accuracy, this study suggests it does not come at the cost of significantly higher interviewer-level noise.
Simple Facts and the Gatekeeper Effect
The source of variance appears to depend heavily on the nature of the question. For simple, factual questions like "birth year," the variance was almost entirely driven by nonresponse error (IIC = 0.025 in the CI group). This indicates a "gatekeeper effect," where certain interviewers are more effective at recruiting specific age cohorts, but once the interview begins, there is little room for measurement error to manifest in such a straightforward variable.
Conclusion and Implications
This research highlights that while interviewers influence data as both recruiters and collectors, their role as data collectors remains the primary source of variance in complex measures. Importantly, this holds true regardless of whether interviewers follow a rigid script or engage in flexible, conversational clarification.
For survey practitioners, these findings suggest that efforts to reduce interviewer variance should continue to prioritize the measurement process and interviewer training regarding question delivery. Furthermore, the lack of increased variance in CI supports its viability as a technique that can improve data quality without necessarily introducing excessive interviewer-level instability. Future research should explore the "liking" hypothesis—whether demographic similarity between interviewers and respondents affects these error sources—and replicate these findings in different cultural contexts.
Summary updated