Language models now run qualitative interviews: they ask the questions, choose the follow-ups, and reflect answers back to the person speaking. Systems such as SparkMe make it possible to interview hundreds of people instead of twenty. That scale only helps if the transcripts still record what each person actually thinks.
There is a specific reason to doubt this. Co-writing studies show that people take up a model's fluent suggestions and end up writing alike. Nobody has checked whether the same thing happens when the model is the one asking the questions. An interviewer that paraphrases ("so it sounds like the main issue is trust?") or asks leading follow-ups could plant its own concepts in many participants. The result would be a dataset that looks diverse but partly echoes the interviewer.
This proposal makes two contributions. The first is a way to measure interviewer-originated content: the share of what participants say that first entered the conversation through the interviewer's words rather than their own. The second is a randomized study with human participants that tests whether automated interviewers raise that share and narrow the range of views across a cohort, compared with an unprobed survey and with trained human interviewers.
Core Claim
The claim is that adaptive LLM interviewers homogenize qualitative data through a measurable channel, which we call framing injection. The interviewer introduces a concept, the participant adopts it, and the same concept then recurs across many participants because one model with a fairly fixed set of default framings interviewed all of them.
If the claim holds, two things should be true. First, the share of each participant's key points that trace back to an interviewer turn should be higher under LLM interviewers than under human interviewers. Second, diversity across participants (unique key points, coded themes) should fall even while topic coverage rises.
We also test a targeted fix. It is a check that runs before any follow-up question is sent. It rejects follow-ups whose substantive concepts do not already appear in the participant's own speech or in the fixed topic guide. This is the non-directive probing that qualitative-methods training teaches human interviewers ("you said X; tell me more about X").
The measurement is the main contribution, and it is useful even if the fix turns out to be unnecessary. As the section on alternatives explains, a well-prompted interviewer may already avoid most of the problem. The study is designed to show whether it does.
Why this gap and not a neighboring one
Three lines of prior work motivate the claim.
Co-writing. Does Writing with Language Models Reduce Content Diversity? logged keystrokes from 38 writers. It found that essays written with an instruction-tuned model lost lexical and argument diversity. The text the humans wrote themselves stayed diverse, so the homogenization came from writers accepting model suggestions. Our design borrows that study's key move, separating text by origin, and applies it to dialogue turns.
Lasting effects. Human Creativity in the Age of LLMs found that exposure to model help left participants producing more similar ideas in later unassisted rounds. This is why we add a delayed follow-up.
Default framings. The Homogenization Problem in LLMs showed models pulling open-ended generations toward a single conventional default. Beyond the Chat reported that conversational interfaces anchor users on examples the model offers. Together they suggest that interviewer follow-ups will carry the model's default framings.
One paper constrains how we measure. Evaluating the Diversity and Quality of LLM Generated Content argues that preference-tuned models are more diverse once outputs are filtered for validity. For that reason we score diversity only over key points that blind coders judge relevant to the topic guide, so that off-topic variety does not count as richness.
The closest systems work is SparkMe. It scores interviews on topic-guide coverage, the number of emergent insights, and the number of turns. It never asks where those insights came from. Our study keeps its objective and adds that missing variable.
Two other seed papers mark the edges of this project. SAGE wants simulated people grounded in rich individual interviews. Interviews contaminated by the interviewer would undermine that plan, so measuring contamination comes first. PersuasionTrace tracks belief movement turn by turn in persuasive dialogue. We bring its process-level view into a setting where the interviewer is supposed to exert no persuasive force, and we treat any measurable movement as contamination.
Research Program
Origin attribution. The core tool attributes each idea to its source in the transcript.
- Every turn, from both interviewer and participant, is split into key points using an extraction and clustering approach close to Padmakumar and He's.
- Each participant key point is linked to the earliest turn in which an equivalent concept appeared. Equivalence is judged by an LLM classifier and checked against human annotation on a calibration set.
- A key point counts as interviewer-originated if its first appearance is in an interviewer turn and is not in the shared topic guide.
- The per-participant injection rate is the fraction of that participant's relevant key points that are interviewer-originated.
- Cohort homogenization is measured with:
- the number of unique relevant key points per 100 participants;
- mean pairwise similarity between participants' key-point sets;
- the number of themes found by blind human coders working from a fixed sample of transcripts.
Validation. Before the main study we confirm that the attribution tool detects planted framings. In this positive control, a scripted interviewer inserts a specific rare concept into one paraphrase. The tool should detect that concept's later uptake at well above chance, and it should not flag the concept when the participant raised it first.
The injection check. This is a lightweight module placed between SparkMe's planner and the participant. Before a follow-up is sent, it extracts the follow-up's content concepts. If any concept is absent from both the participant's history and the topic guide, it asks the planner to rewrite the question using only the participant's own terms. Everything else in the system is unchanged: the agenda tracking, the simulated conversation rollouts used for planning, and the utility function.
Simulation fidelity. As a secondary question, we run the same arms against LLM personas of the kind SparkMe uses for benchmarking. We then check whether simulated interviewees reproduce the human injection rates. PersuasionTrace reports that vanilla simulated targets score 64 on human-likeness against a human reference of 80. If the same gap shows up here, cheap simulator benchmarks would hide the effect we are measuring.
Baselines
Unprobed survey (the "do nothing" reference). This is an open-ended web survey that asks the topic-guide questions verbatim and never follows up. It has close to zero injection by construction. It shows how diverse people are when nothing anchors them, and it shows how shallow their answers are without probing. An LLM interviewer "wins" against this arm only if it adds depth, meaning more relevant key points per participant, without lowering per-participant diversity below the survey's level.
Trained human interviewers. Practitioners who have been trained in non-directive probing run interviews on a smaller subsample using the same guide. This is the realistic gold standard for depth with low contamination. We pre-register the human injection rate as the target to match.
SparkMe as published (the strongest existing method). It is run with the same backbone model, the same topic guide, and the same turn budget as our variants. On its synthetic persona benchmark it reports +4.7% topic-guide coverage over the best baseline, MimiTalk. We found no reported result on Lacuna for any measure of response homogenization or interviewer injection in SparkMe or any other automated interviewer. That number is exactly what this study supplies.
SparkMe with a neutrality prompt. This is SparkMe with explicit instructions to avoid leading questions and paraphrases that introduce new ideas. It is the cheapest plausible fix and the main threat to our intervention.
SparkMe with the injection check. The difference between this arm and the published SparkMe arm isolates the contribution of the check. The difference between this arm and the neutrality-prompt arm shows whether an explicit verifier does anything a prompt cannot.
Fairness and success criterion. All automated arms share the model, the decoding settings, and an equal prompt-development budget, tuned on the same pilot of 15 participants. The injection-check arm counts as successful only if it meets both conditions:
- it brings the injection rate to within the human-interviewer range and recovers cohort diversity to at least survey level;
- its topic coverage stays within a pre-registered margin of published SparkMe.
Why not just…?
Why not just tell the interviewer not to lead? This is a real threat, and we test it directly as a baseline. Modern instruction-following models may comply well. If the neutrality-prompt arm matches the injection-check arm, we will report that the problem is solved by prompting and that the contribution is the measurement. Nothing in the file shows prompting fails here. What we have is indirect: in the co-writing study, homogenization came from content that writers chose to accept, which suggests that how fluent a framing is, not only how leading it is, drives uptake. A prompt targets leading questions and may miss neutral-sounding paraphrases that still carry new concepts. The comparison will settle the question.
Why not just use an open-ended survey and avoid the issue? A survey has no anchoring, but it cannot probe. SparkMe's motivation and its 70-person expert evaluation both point to probing as the source of depth. The survey arm measures what that depth costs: if the LLM interviewer adds key points mostly by injecting them, the survey is the more honest instrument, and our data will say so.
Why not just measure this with simulated personas, which is far cheaper? PersuasionTrace's 64-versus-80 human-likeness gap for vanilla simulated targets suggests that simulators may not show human uptake behavior at all. Our simulation-fidelity comparison tests this directly. If simulators do reproduce the human injection rates, later work can safely use them, and that would itself be a useful finding.
Why not just raise sampling diversity, such as temperature or multiple interviewer personas? That would make the interviewer's questions more varied. It would not stop each question from introducing concepts that are not the participant's own. Varied injection is still injection, and the attribution tool would count it, so we do not treat it as a main arm.
Proposed Experiments
Main study. About 300 working professionals discuss how AI affects their work, which mirrors SparkMe's domain. They are randomized across four arms: survey, SparkMe, neutrality prompt, and injection check, with about 75 per arm. A separate stratified sample of about 40 participants is interviewed by humans.
Primary outcomes: - the injection rate; - cohort homogenization, measured by unique relevant key points, mean pairwise similarity, and the number of human-coded themes.
Secondary outcomes: - topic coverage and emergent insights, scored as in SparkMe; - participant-rated agency, meaning whether the account felt like their own.
Lasting effects. Following the design of the creativity trials, all participants answer an unassisted written question on the same topic one week later. We then test whether interviewer-originated concepts persist in their writing.
Downstream analysis. As a smaller check, transcripts from each arm go through LLM-in-the-loop thematic coding in the style of Dai et al. and through codebook-prompted deductive coding as in Xiao et al.. This tells us whether analysis by an LLM compounds homogenization introduced during elicitation. Xiao et al. report codebook coding reaching κ = 0.61 on semantic categories, so human coders remain the reference for theme counts. This follows the reasoning in Strategic Level of Automation Selection, which keeps human judgment for interpretive synthesis.
Decisive checks: 1. The attribution tool must pass the planted-framing positive control before any human data is analyzed. 2. The central hypothesis fails if published SparkMe's injection rate is no higher than the human interviewers'. In that case we report that automated interviewing does not homogenize through this channel, which is a useful negative result for anyone scaling qualitative research. 3. The injection check is justified only if it beats the neutrality prompt on injection rate at matched coverage. Otherwise the recommended practice is simply the prompt.
Workshop authors to connect with
- A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing: Jared Moore, Noah Goodman, Nick Haber, Max Kleiman-Weiner
- For Questions of Ought, AI Could Use Some SAGE Advice: Smitha Milli, Ratip Emin Berker, Sonja Kraiczy, Claudia Shi, Jack Kussman, Avinandan Bose, Himaghna Bhattacharjee, Edith Elkind, Ariel D. Procaccia, Maximilian Nickel
- SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery: David Anugraha, Vishakh Padmakumar, Diyi Yang
References
- SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery
- Does Writing with Language Models Reduce Content Diversity?
- Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking
- The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety
- Beyond the Chat: Executable and Verifiable Text-Editing with LLMs
- Evaluating the Diversity and Quality of LLM Generated Content
- For Questions of Ought, AI Could Use Some SAGE Advice
- A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing
- LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
- Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding
- Structuring Human-AI Productive Interdependence by Strategic Level of Automation Selection for Qualitative Inquiry