Keeping Every Participant's Voice: Participant-Wise Pareto Search for LLM Thematic Analysis

Thematic analysis is the standard way qualitative researchers turn interview transcripts into a small set of themes, the recurring ideas that summarize what people said. Large language models now do a first pass in minutes instead of the 30 to 60 hours a human team needs. They share one known failure: they converge on what most participants said and quietly drop the one parent, patient, or user whose experience was different. In qualitative work that outlier is often the finding. This proposal treats each participant as a separate test the codebook must pass. A codebook here means the list of named codes, each with a definition and supporting quotes. The search keeps alive every candidate codebook that represents at least one participant better than any rival, and it uses the specific unrepresented quotes as feedback for rewriting.

This borrows the core move of Pareto-based evolutionary prompt search, where a candidate survives if it is best on some problem instance rather than best on average. It moves that move into a setting where the "instances" are people and the score is whether their own words are accounted for. Two sentences for a colleague: current LLM thematic analysis pipelines optimize an averaged or judged quality score, which favors majority themes. We instead run an evolutionary search whose selection rule protects any codebook that best represents some individual participant, under a fixed codebook size, and we test whether this recovers the rare codes human coders found without losing overall agreement.

Core Claim

The claim is that per-participant diversity preservation recovers low-frequency human codes at no cost to overall agreement with human coders. This kind of preservation keeps codebooks that win on at least one participant's transcript. We expect it to beat three alternatives: aggregate-objective Pareto search, which trades off coverage against distinctiveness and similar summary scores; scalar refinement, which optimizes one judged score; and a simple max-min fairness objective. The ingredient that is new relative to the closest work is the unit of diversity. It is not spread along objective axes. It is which people a codebook can speak for.

Three existing multi-agent pipelines motivate the proposal by showing both the promise and the gap. TAMA uses generation, evaluation, and refinement agents to edit themes with add, split, combine, and delete operations, judging them on coverage, actionability, distinctiveness, and relevance. On a focus-group study of 42 parents of children with a congenital coronary condition (AAOCA), it raised hit rate against human themes from 83% to 92%. It resolves trade-offs between those criteria inside agent prompts. Auto-TA adds identity-conditioned agents and an optional reinforcement-learning loop (PPO, a standard policy-gradient method) over expert ratings, which collapses several quality criteria into one reward. SFT-TA fine-tunes agents on 300 examples and fixes the output at exactly 12 themes. Its authors report run-to-run instability. None of the three keeps a population of competing codebooks, and none checks representation participant by participant.

On the optimization side, GEPA supplies the method. It evolves prompts by having an LLM reflect on failure traces, and it keeps a frontier of candidates that are each best on some training instance. It outperformed GRPO by 6 points on average with 35x fewer rollouts. optimize anything generalizes this to arbitrary text artifacts guided by diagnostic "side information," meaning a textual explanation of why a candidate failed. Both assume an automatic evaluator such as unit tests or answer checking. Neither has been applied to codebooks, where the evaluator is itself a judgment.

Pareto Prompt Optimization shows that scalarized prompt optimization collapses onto narrow compromise solutions, and that dominance-based training spreads candidates along the trade-off curve. Its objectives, however, are a handful of global scores such as accuracy and fluency. That is the aggregate-objective design we test against, not the one we propose.

The many-objective theory literature constrains the design. Runtime analysis of NSGA-II shows that with many objectives nearly every solution becomes mutually non-dominated, so selection pressure disappears. With 15 participants treated as 15 objectives, plain non-dominated sorting would keep everything. We therefore use GEPA's "best on at least one instance" rule, sampling parents in proportion to how many participants they win. That rule stays selective however many instances there are. We use it in place of NSGA-II dominance or NSGA-III reference directions, which were designed for few objectives.

Research Program

A candidate is a codebook of at most $K$ codes. Each code has a name, a one-sentence definition, and quotes it claims to cover. Before search begins, an LLM splits each participant's transcript into meaningful statements. The per-participant score is

$$s_c(p) = \frac{1}{|U_p|}\sum_{u \in U_p} \mathbb{1}[\exists\, k \in c : J(k,u)=1],$$

where $U_p$ is the set of statements from participant $p$, $c$ is the codebook, $k$ ranges over its codes, and $J(k,u)$ is an LLM judge's decision that code $k$ captures statement $u$. The judge must name the code and quote the span, so its decisions can be audited.

The judge never sees human codes. The search is therefore reference-free and inductive, and all human annotations are held out for evaluation.

The size cap $K$ is a hard constraint, not an objective. Without it, coverage rises trivially as codes are added, and the largest codebook would win on every participant. A code is accepted only if its quote overlap with existing codes stays below a distinctiveness threshold, measured by Jaccard similarity as in TAMA.

Selection. A codebook stays on the frontier if it has the highest $s_c(p)$ for at least one participant $p$, ties included. Parents are sampled in proportion to the number of participants they win.

Mutation. A reflective LLM receives the parent codebook and the side information for one sampled participant. That side information is the participant's uncaptured statements, with quotes, and it serves as a machine-readable version of the audit trail reflexive analysts keep. The LLM proposes edits drawn from TAMA's operations: add, split, combine, delete.

Merge. A merge operator takes codes from two frontier parents that win on different participants and compresses the union back to $K$ codes.

Output. The system returns more than a single winner. It returns the frontier itself. It also returns a consensus codebook, chosen as the frontier member with the best worst-participant score, and a list of "minority codes" that appear only in some frontier members, each tagged with the participants who motivated it. The analyst decides what to keep. This responds to the concern that full automation removes interpretive agency.

Sweeping $K$ yields a granularity curve. Qualitative Code Suggestion found that human coders differ sharply between abstract and granular styles, and released CVDQuoding with annotations from two coders. We predict that small-$K$ frontiers match the abstract coder and large-$K$ frontiers match the granular coder. Making granularity an explicit dial would replace SFT-TA's fixed choice of 12 themes.

Baselines

The naive reference point is a single long-context prompt. A strong model sees all transcripts and is asked for $K$ distinct, comprehensive codes with supporting quotes. It runs with the same base model as every other method, and we draw five seeds to expose its variance. Beating it is the minimum bar. If we fail to, the search is not worth its cost.

The strongest existing method is a TAMA-style evaluate-and-refine loop, reimplemented as TAMA describes it. It uses the same base model and the same edit operations, and it runs for as many LLM calls as our search consumes, so it is not short-changed on compute. TAMA reports a 92% hit rate and 29% inter-theme Jaccard similarity on AAOCA. We found no reported result on Lacuna for TAMA or any theme-level method on CVDQuoding. The only CVDQuoding number is Precision@4 above 0.5 for passage-level code suggestion in Qualitative Code Suggestion, which is a different task. On CVDQuoding we must therefore produce the baseline numbers ourselves. To count as beating TAMA, we need to match or exceed its overall matched recall within noise and to improve rare-code recall by a margin exceeding the seed-to-seed spread.

Best-of-$N$ with a participant-wise filter isolates whether evolution matters at all. It samples $N$ codebooks independently at the same total budget and returns the participant-wise frontier of that pool.

Scalar max-min refinement uses the same reflective mutation operator but keeps a single incumbent. That incumbent is scored by its worst participant, $\min_p s_c(p)$, plus a small mean term to break ties. This is the ablation most likely to embarrass the proposal, because it targets minority voices directly without any population.

Aggregate-objective Pareto search, in the style of Pareto Prompt Optimization, keeps a frontier over four summary scores: mean coverage, distinctiveness, actionability, and size. It tests whether the participant-wise unit is the key ingredient.

A final ablation removes side information and gives the mutation operator only scores, as a test of the claim from optimize anything that diagnostic text drives efficiency.

All methods share the base model, the judge, the statement splitting, the cap $K$, and a fixed budget counted in LLM tokens. Each prompt-based method gets the same small tuning allowance on a single development transcript that is then excluded from evaluation.

Why not just…?

Why not just ask a long-context model for the themes in one shot? The surveyed evidence says single prompts are phrasing-sensitive and shallow (Position paper). In SFT-TA, plain GPT-4o trailed a structured pipeline by 10.3% fuzzy match and 15.3% cosine similarity (SFT-TA). Those gains, however, are over aggregate agreement, not rare codes. Our one-shot baseline with five seeds measures directly whether a single prompt already surfaces the minority codes.

Why not just optimize the worst-covered participant as a single score? This is the real threat, and we treat it as one. Max-min scalarization may capture most of the benefit. Our reason to expect it to fall short is that a single incumbent must cover every participant with only $K$ codes. Improving one participant often means evicting a code another participant needs, and a lone-incumbent search cannot hold both options open for later merging. If the max-min refinement baseline matches our rare-code recall at equal budget, we will report that the population is unnecessary and recommend the cheaper method.

Why not just sample many codebooks and filter afterward? If independent samples already cover the space, evolution adds cost without value. GEPA found reflective evolution far more sample-efficient than undirected alternatives, but not on qualitative data. The equal-budget best-of-$N$ comparison settles this. We will also report how the gap scales as $N$ grows.

Why not just run topic modeling or embedding clustering? Clustering spreads clusters in embedding space, but it cannot split compound ideas or write definitions, and lexical or embedding methods struggle with abstract codes (Qualitative Code Suggestion). We include a clustering run over sentence embeddings with clusters labeled by an LLM, mainly as a floor for rare-code recall.

Proposed Experiments

Data. The primary dataset is CVDQuoding: 15 interviews with women at risk of cardiovascular disease, 1,175 annotations from two coders with different styles. Its per-transcript structure supplies participants as instances, and its two coders supply abstract and granular references. Secondary datasets are the Password Manager responses (280) and the Music Shuffle responses (35) from LLM-in-the-loop, with each response treated as a participant, if their authors release them. AAOCA would be used only through collaboration with its authors, given privacy limits, so that TAMA's published numbers can be compared directly.

Metrics. Following the evaluation proposal in the Position paper, we use maximum-weight bipartite matching between generated and human codes to handle many-to-many correspondence. Validity is measured as matched recall and precision against each coder separately. The headline measure is rare-code recall: recall restricted to human codes that appear in at most two transcripts. Representation is measured as worst-participant recall against human codes. Reliability is measured as agreement of codes across five seeds using Krippendorff's alpha. Distinctiveness is measured as inter-code Jaccard similarity.

Decisive checks. The first check is whether participant-wise search raises rare-code recall over the TAMA-style loop and max-min refinement at equal budget without lowering overall matched recall. Second, does it do so over aggregate-objective Pareto search? That would show the participant unit, not the population alone, is responsible. Third, does sweeping $K$ reproduce the abstract-versus-granular split, with each coder's codebook best matched at a different $K$? Fourth, does keeping a frontier improve seed stability relative to single-incumbent methods, addressing the instability SFT-TA reports?

Human check. A small study with two qualitative researchers on CVDQuoding asks whether flagged minority codes are substantive findings or judge artifacts. This guards against the judge rewarding vague codes that technically "cover" outlier statements.