Checking the Quotes: Word-Level Quality Estimation for Evidence in LLM-Assisted Thematic Analysis

When qualitative researchers use a language model to code interview transcripts, the model does two things. It assigns a theme, such as "distrust of cloud storage," and it cites a passage from the transcript as evidence for that theme. Current evaluations check only the first step: did the model's label agree with a human's? Nobody checks whether the quoted words actually support the label, or whether the quote cuts off the clause that makes it relevant, or drags in surrounding text that says something else. Machine translation has a mature tool for this kind of question. Word-level quality estimation predicts, without any reference answer, which individual words of an output are wrong. We propose to adapt that tool to thematic evidence. A small open model would tag every token of a transcript as properly cited support, cited but irrelevant, or needed but left out, given the code it is meant to support. The whole pipeline would run on local open-weight models, so sensitive transcripts never leave the institution.

This matters because the speed argument for LLM-assisted coding is real. The clinical thematic analysis position paper, which surveys 56 studies, reports initial coding in under 10 minutes against 5 to 8 hours by hand. The same survey finds the evaluation landscape fragmented: 40% of studies rely on unblinded human review, 27% on n-gram overlap metrics, and 74% use OpenAI models, even though clinical narratives often cannot be sent to third-party APIs. A coder that is fast but whose evidence cannot be checked token by token simply moves the labor of verification onto the analyst.

Core Claim

Our central claim is that the dangerous errors in machine-assisted coding are mostly not invented quotes. Those can be caught by checking whether the quote appears verbatim in the transcript. The dangerous errors are authentic quotes that are misattributed to a code or wrongly scoped. A token-level estimator trained on synthetically corrupted evidence spans can localize these errors better than either a passage-level "does this support the code" score or the coding model judging its own output. The estimator should be scored against the distribution of judgments from several human coders, not against a single majority-vote span. The new element is the target itself. Word-level quality estimation has always tagged generated translations. Here it tags extracted evidence relative to a code definition, and its errors are scoped against plural human readings.

Research Program

The work has three parts.

The first part is a coding pipeline built entirely from open models. We will reimplement the four-stage human-in-the-loop procedure of LLM-in-the-loop. That procedure has four steps: the analyst writes worked examples, the model proposes codes, the model clusters codes into themes under analyst correction, and both apply the final codebook. The original used GPT-3.5. We will use fully open models such as OLMo 2 32B, whose training recipe is transparent (2 OLMo 2 Furious), and a Llama 3.1 70B model. We change one thing: every code assignment must come with a character-offset evidence span. The split between a generative model that proposes and a separate checker that verifies follows the approach of HerO. There, a pipeline of open Llama models with specialized stages placed second on the AVeriTeC fact-checking benchmark, behind only a GPT-4o system.

The second part is the evidence estimator. Its input is a code definition, the transcript window, and a cited span. Its output is a tag for each transcript token. Tokens inside the span that support the code are OK. Tokens inside the span that do not support it are over-scoped. Tokens outside the span that the code depends on are under-scoped, the analogue of the omission tags used in translation quality estimation. The model is an XLM-R Large-style encoder with a token-tagging head and a span-level support score, trained with the joint loss from NJUNLP's WMT 2023 system. That loss combines mean-squared error with a margin ranking loss for the scalar score and token cross-entropy for the tags, and spans are derived by thresholding and merging token probabilities.

Training data comes from corruption, as in NJUNLP's synthetic error generation. We start from human-cited spans. We then shrink or extend their boundaries, slide them to adjacent sentences, and reattach them to sibling codes in the codebook. Because every corruption is known, it yields exact token labels. We will take a gradient-attribution signal over the coder's own evidence choice as a secondary feature, borrowing the Integrated Gradients approach used by He et al. to find under-translation. This feature is kept only if ablation shows it helps.

The third part is a gold set and harness built around disagreement. Taking seriously the argument of The Consensus Trap that majority vote erases legitimate minority readings, we will have $K$ trained coders independently mark supporting spans for each code assignment. For token $t$, we define the soft target $p_t$ as the fraction of the $K$ coders who mark $t$ as support. An estimator's predicted support probability $\hat p_t$ is then scored with a token-averaged Brier score:

$$ B = \frac{1}{T}\sum_{t=1}^{T} (\hat p_t - p_t)^2, $$

where $T$ is the number of evaluated tokens. We also report Matthews correlation coefficient and span F1 against the majority span, so results stay comparable to translation quality estimation work.

HUME found that a high model score on a task where humans disagree, such as ArXiv clustering, can reflect memorized labeling quirks rather than understanding. Following that lesson, the harness reports human-human agreement next to every model score. It also flags items with low inter-coder agreement instead of rewarding a model for matching one coder. The harness fixes splits, seeds, prompts, and thresholds in the way that NerfBaselines and BARS did for their fields. Both papers found that unstandardized pipelines manufactured apparent gains, and BARS found that properly tuned baselines erased several claimed improvements.

Baselines

The naive reference point is to trust every cited span after a verbatim string-match check. This removes fabricated quotes and treats everything else as correct. Its score on misattribution and scoping errors is, by construction, the base rate of those errors in the coder's output. We will report that rate directly, because if it is small the whole proposal is moot.

The strongest existing method for the coding stage is the GPT-3.5 human-in-the-loop pipeline of LLM-in-the-loop. It reports a Cohen's κ of 0.87 on Music Shuffle and 0.81 on Password Manager, where the original human-human agreement was 0.77. Our open pipeline must match these κ values on the same datasets, with the same number of analyst-written examples and the same analyst feedback rounds, before its evidence quality is even worth discussing.

For the evidence-checking stage there is no reported result on Lacuna for token-level verification of thematic evidence spans. The nearest reported number comes from a different task: NJUNLP's word-level MCC of 29.7 on WMT 2023 English-German translation (NJUNLP). The WMT 2024 findings report span F1 between 0.15 and 0.48 for error-span detection in translation (WMT 2024 QE findings). These figures calibrate expectations but are not targets.

The competitors we will actually run on our task are three:

  • Passage-level support scorer. The same encoder, trained on the same corrupted data, outputs one support score per cited span.
  • Self-judging generator. The open generative coder is prompted with few-shot examples to mark unsupported or missing words itself.
  • Frontier reference. GPT-4o is prompted the same way, run only on the non-sensitive Music Shuffle and Password Manager data.

All three get the same training and development splits, the same threshold-tuning budget on development data, and the same token budget for prompting.

Beating them means the following. The estimator should produce lower Brier score and higher MCC than both the passage-level scorer and the self-judging generator, by margins exceeding seed-to-seed variance across five seeds. It should also catch more planted misattributions at a fixed reviewer budget, defined as the top 20% of flagged spans.

Two ablations isolate the key ingredients. One trains without boundary-shift corruptions, keeping only code-swap corruptions; this tests whether scoping errors need explicit supervision. The other evaluates against majority-vote spans instead of soft targets; this tests whether system rankings change.

Why not just…?

Why not just check that quotes appear verbatim? Verbatim checking is cheap and kills fabricated quotes, and we include it as the floor. It cannot detect an authentic sentence attached to the wrong code, or a quote clipped before the qualifying "but". These are exactly the errors that let a plausible theme rest on misread evidence. Our first measurement will estimate how common these errors are in open-model coder output. If they turn out to be rare, the honest conclusion is that string matching suffices, and we would report that.

Why not just score the whole passage instead of individual tokens? This is the most serious threat. A passage-level "supports / does not support" score may catch most misattributions, and the WMT 2024 findings show that fine-grained span boundaries are where systems are weakest. We run exactly this scorer on identical data. If it matches the token-level estimator on misattribution catch rate and differs only on scoping errors that human coders themselves disagree about, as measured by low agreement on boundaries, we would adopt the simpler scorer. Token tagging would then be kept only for the subset of items where boundaries are reliably agreed.

Why not just have GPT-4o do it? On non-sensitive data it may well be better, and we report it as a ceiling. The privacy argument from the clinical position paper makes it unusable for the transcripts that motivate this work. Open models have matched or beaten GPT-4o on specialized tasks such as long-form medical question answering (Hosseini), and WMT 2024 found that generative models of both kinds trail dedicated encoders at span localization. The relevant question is therefore whether a small local estimator closes the gap, not whether a frontier model wins.

Why not just ask the coding model to verify its own spans? Self-verification costs nothing extra. However, it inherits whatever misreading produced the error, and the granularity gap documented at WMT 2024 suggests generative models localize boundaries poorly. The self-judging generator baseline tests this directly.

Proposed Experiments

Experiment 1: open-model coding parity. We first reproduce coding agreement with open models on Music Shuffle (35 responses) and Password Manager (280 responses), comparing κ against the 0.87 and 0.81 reported for GPT-3.5. This establishes that open models are viable coders before we test their evidence.

Experiment 2: annotation. We collect independent span annotations from four or five trained coders for every code assignment in both datasets. Both are small, so the gold set will hold hundreds rather than thousands of assignments. We will add a non-public interview corpus under ethics approval to test generalization, and report all results with bootstrap confidence intervals.

Experiment 3: estimator comparison. We train the estimator on corruptions of one dataset and evaluate on the other, so it cannot memorize a single codebook. It is compared with all baselines on Brier score, MCC, span F1, and planted-error catch rate.

Experiment 4: disagreement-aware scoring. We check whether ranking systems by soft-target Brier score versus majority-span F1 changes their order. If the order changes, the evaluation choice matters. If it does not, majority vote is defensible for this task.

Experiment 5: user study. This is a small study modeled on QE4PE, whose 42 translators caught 16–20% more critical errors with highlights but showed no speedup. Analysts review coded transcripts containing planted misattributions in three conditions: no highlights, passage-level flags, and token-level highlights. We measure catch rate, minutes per transcript, and self-reported burden. The decisive outcome is whether token highlights raise catch rate without the extra verification time QE4PE observed. If they raise catch rate only at a time cost that passage-level flags avoid, the practical recommendation is passage flags, with token highlights reserved for audits.