When researchers use large language models (LLMs) to find themes in interview transcripts, they face a question with no principled answer: how many times should the model code the data, and how much data must be coded, before no important theme is still missing? Current practice picks a fixed number. One study reruns each model six times per transcript and keeps themes that recur (Multi-LLM Thematic Analysis with Dual Reliability Metrics). Others run once at zero temperature and truncate datasets that do not fit the budget (LLM-in-the-loop). Qualitative researchers have long called the stopping point "thematic saturation," the moment new data stops producing new codes. They judge it by eye. We propose to turn saturation into a stopping rule the researcher can tune, in the spirit of approximate computing, the systems practice of trading a bounded, user-specified error for less computation. The rule stops sampling when a statistical upper bound on the "missing mass" falls below a tolerance $\epsilon$ that the researcher chooses. The missing mass here is the probability that the next coding pass would produce a theme not yet seen.
The statistics already exist, but nobody has checked them against real data. Concentration Inequalities for the Missing Mass and for Histogram Rule Error proves that the missing mass concentrates tightly without distributional assumptions. The file describes applying it to practical data-dependent algorithms as unaddressed. The proposal therefore does two things. It builds an adaptive-compute method for LLM thematic analysis, and it tests whether a theoretical bound survives contact with messy, non-independent, embedding-clustered qualitative codes. The first contribution fills a gap in that literature. The second addresses the validation gap in the statistical theory. Either result is informative.
Core Claim
A Good-Turing-style estimate of unseen theme probability, plus a concentration slack term, can serve as an early-stopping controller for LLM thematic analysis. Good-Turing estimation infers how much probability belongs to unseen categories from how many categories have been seen exactly once. The controller works at two levels. At the first level it decides how many stochastic LLM passes to run over a transcript. At the second it decides how many responses or transcripts to code before declaring saturation. We claim three things. First, the controller will spend substantially fewer passes than a fixed six-run ensemble while recovering the same consensus themes. Second, it will be calibrated, meaning the realized missing mass at the stopping point exceeds $\epsilon$ in at most roughly a $\delta$ fraction of stops. Third, the tolerance $\epsilon$ gives researchers an explicit accuracy-for-compute setting, which no current tool offers.
Concretely, after $m$ coding draws the controller clusters codes into equivalence classes. It uses the rule from Jain et al.: two codes are the same class if their embedding cosine similarity exceeds 0.70. It then counts $N_1$, the number of classes seen exactly once. It stops when
$$ \hat{U}_m = \frac{N_1}{m} + c\sqrt{\frac{\ln(1/\delta)}{m}} \le \epsilon , $$
where $N_1/m$ is the Good-Turing estimate of the missing mass, $\delta$ is the allowed failure probability, and $c$ is a constant. The constant comes from combining the McAllester-Ortiz upward-deviation bound for the missing mass, which has the form $e^{-m\epsilon^2}$, with a matching concentration bound for the estimate itself. That combination is why the slack shrinks as $1/\sqrt{m}$. We will derive $c$ explicitly and also report a conformally tuned empirical variant.
Research Program
The work has three parts.
Within-transcript stopping. A "draw" is one full LLM coding pass over a transcript at nonzero temperature. Conditional on the transcript and prompt, draws are independent and identically distributed, so the theory's assumptions hold cleanly. This is the replacement for Jain et al.'s fixed six runs. It also turns their proposed but never implemented "Adaptive Run Configuration" into an algorithm with a guarantee.
Across-corpus stopping. A draw is the coding of one new response or transcript. Here independence genuinely fails: interviews differ by participant, and ordering matters. We randomize coding order to recover exchangeability across orderings. We then measure how much calibration degrades when the theory's assumptions are only approximately met. This is the empirical-validation half of the project, and a negative result is publishable.
Sensitivity to what counts as a "new theme." Missing mass depends on how codes are grouped, and that grouping is a modeling choice, not a fact. We will sweep the cosine threshold and the embedding model to see how stopping decisions shift. We will also stratify missing mass by segment complexity. The motivation is the "complexity flip" in Text Annotation via Inductive Coding: LLMs match humans on literal statements but trail them on abstract ones. That suggests the unseen themes concentrate in hard segments, so a stratified bound could stop early on easy strata and keep sampling on hard ones.
How the guarantee is framed. The guarantee concerns the model's own distribution of codes, not an objective truth. This matters given the reflexivity critique in qualitative methodology and the very low human agreement in open coding: Krippendorff's alpha of 0.2 in Parfenova et al. We therefore present the output to researchers as a coverage statement ("fewer than $\epsilon$ of further passes would yield a theme you have not seen"). It is not a claim that the theme set is correct. We also measure separately how the stopped theme set compares with human gold codes.
We will use open-weight models: Mistral-7B and Llama3-8B, which Parfenova et al. found competitive after roughly 100 fine-tuning examples. Local models address the privacy concern with clinical transcripts. We will add one commercial model to allow comparison with published agreement numbers.
Baselines
Single deterministic pass. The naive reference point is one pass at temperature 0, the setting used in LLM-in-the-loop. That paper reports Cohen's κ of 0.81 on Password Manager and 0.87 on Music Shuffle against human coding. This baseline defines the minimum compute. Our rule should cost at most a small multiple of it while recovering themes that a single pass misses.
Fixed six-run ensemble. The strongest existing method is the Monte Carlo ensemble of Multi-LLM Thematic Analysis: six runs at temperature 0.7, with a consensus threshold of 50% of runs and cosine similarity 0.70. For Gemini 2.5 Pro it reports κ = 0.907, cosine similarity 95.3%, and six consensus themes, at about $0.15–$0.20 per transcript. We reimplement it with identical prompts, models, temperature, and clustering. Only the number of passes differs, and it is compared at matched token budgets. Beating it means reaching the same consensus theme set, and κ within 0.02 of its κ, using fewer passes on average. It also means reporting calibrated coverage, which the fixed ensemble cannot provide.
Heuristic "no new codes in the last k draws." This formalizes the practitioner's saturation rule. It gets the same model, clustering, and draw stream, and we tune $k$ on held-out data with the same tuning budget we give $c$.
Oracle stopper. An oracle that stops at the first $m$ where the realized missing mass falls below $\epsilon$ gives the compute lower bound. The realized missing mass is measured against a reference distribution built from 30 passes per transcript.
Ablations. The key ablation removes the slack term and uses the bare Good-Turing estimate. It isolates whether the concentration bound, as opposed to the point estimate, is what delivers calibration.
Why not just…?
Why not just run a fixed number of passes, as Jain et al. do? A fixed count spends the same compute on a transcript whose themes stabilize after two passes and on one that needs ten, and it says nothing about what was missed. Jain et al. evaluated one transcript, so there is no evidence that six generalizes. Our compute-versus-realized-missing-mass curves across the file's datasets will show whether any fixed count is simultaneously efficient and safe. If a fixed six turns out to be both efficient and calibrated on every dataset, the controller's value shrinks to certification alone, and we will report that.
Why not just stop after k consecutive draws with no new code? This is the real threat. It is simple, and with a well-tuned $k$ it may match our pass counts. The difference is that $k$ has no stated error meaning, and a $k$ tuned on one corpus may be miscalibrated on another. The decisive test trains $k$ on one dataset and evaluates calibration on the others, and does the same for $c$. If the heuristic transfers as well as the bound does, we will say so. The contribution then becomes empirical evidence about when saturation heuristics are trustworthy, rather than a new controller.
Why not just use the Good-Turing point estimate without the bound? For small $m$ the point estimate is noisy and tends to stop too early. That is exactly the regime where a stopping rule saves compute. The slack ablation measures this directly as the gap between the target failure rate $\delta$ and the observed failure rate.
Why not just use topic models or BM25 instead of LLM passes? These are cheaper, but Automating Qualitative Data Analysis with Large Language Models documents the "meaning gap" of LDA-style models. Qualitative Code Suggestion shows BM25 is competitive only on novel-code detection, not on suggesting codes. The stopping rule does not depend on the choice of coder, though. We will apply it to a BM25 code-suggestion stream on CVDQuoding to check that the rule is useful regardless of coder.
Proposed Experiments
Datasets. Within-transcript stopping uses CVDQuoding (15 cardiovascular interviews, 1,175 dual annotations), Music Shuffle (35 responses), and Password Manager (280 responses). We will run the full 280, which LLM-in-the-loop had to downsample to 100 because of context limits. We will also use the art-therapy transcript if it can be obtained. For each unit we build a 30-pass reference distribution at temperature 0.7.
Across-corpus stopping uses CVDQuoding's sequential interviews and the 1,000 code-quote pairs from Parfenova et al., each under 200 random orderings. There, missing mass is measured against the frequency of human gold codes in the full corpus.
Metrics. We report four things. Compute is measured as passes, tokens, and GPU-seconds. Realized missing mass is measured at the stopping point. Calibration is the fraction of stops that violate $\epsilon$, compared with $\delta$, for $\epsilon$ in {0.02, 0.05, 0.1}. Agreement with human gold codes uses Cohen's κ and cosine similarity, reported so they can be compared with the numbers above.
Decisive checks. The project succeeds if the bound is calibrated in the independent within-transcript setting, and our rule recovers the six-run consensus themes with fewer average passes and no κ loss. The across-corpus results then tell us whether the guarantee survives realistic dependence. A clear failure there, with the violation rate far above $\delta$ even under randomized ordering, is itself the validation result the theory lacks. It would motivate dependence-aware corrections.
On CVDQuoding we will also check whether the stratified, complexity-aware variant concentrates its extra passes on abstract segments. That would be direct evidence that the complexity flip shapes where themes stay hidden.
References
- Multi-LLM Thematic Analysis with Dual Reliability Metrics: Combining Cohen's Kappa and Semantic Similarity for Qualitative Research Validation
- LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
- Concentration Inequalities for the Missing Mass and for Histogram Rule Error
- Text Annotation via Inductive Coding: Comparing Human Experts to LLMs in Qualitative Data Analysis
- Automating Qualitative Data Analysis with Large Language Models
- Qualitative Code Suggestion: A Human-Centric Approach to Qualitative Coding