Remembering Why Themes Were Rejected: Solver-Backed Memory for Multi-Agent Thematic Analysis

Teams of language-model agents can now draft a thematic analysis of interview transcripts in minutes. Thematic analysis is the qualitative-research practice of tagging passages with short labels ("codes") and grouping those codes into themes. In these systems, one agent proposes themes, another critiques them, and a third edits them, and they repeat this for a few rounds. The weak point is that the loop forgets. When a critic says "these two codes do not belong in one theme," that verdict lives only in a prompt. The next round can regroup the same codes, edits can undo earlier edits, and the authors of the leading systems report regression, stagnation, and run-to-run inconsistency. We propose to borrow the main trick of modern SAT solvers, programs that search for assignments satisfying logical constraints. When such a solver hits a dead end, it records the reason as a "learned clause," a short logical rule that stops it from ever entering that dead end again. It keeps those clauses across a sequence of related problems instead of starting cold each time.

Concretely, a SAT solver proposes how codes are grouped into themes, subject to hard requirements such as covering every code and staying within a theme budget. LLM critic agents judge whether each proposed theme is coherent. Every rejection is shrunk to a minimal reason, for example "codes a and b cannot share a theme." That reason is stored as a clause that applies to every theme slot, not just the one that was rejected. Clauses persist across refinement rounds and across newly added transcripts. Each clause is guarded by a switch that lets a human retract it. The claim is that generalized, retained rejections, rather than more agents or more rounds, are what make automated refinement converge. They should also make it reproducible and auditable. The same machinery yields a principled signal of thematic novelty: a new transcript whose codes cannot be fitted into the existing themes without violating retained clauses.

Core Claim and What Is New

Three multi-agent thematic-analysis systems define the current approach, and all three motivate this work by documenting the gap. TAMA pairs generation, evaluation, and refinement agents with a human expert, using ADD/SPLIT/COMBINE/DELETE edits. It reports that an unsupervised LLM judge stagnates or hallucinates changes without human termination. SFT-TA inserts fine-tuned coding agents into a four-stage pipeline with at most three refinement rounds, and it flags poor run-to-run dependability. Auto-TA removes the human and adds persona conditioning and optional reinforcement learning from human feedback. It is "highly prone to theme overlap and regression" during autonomous editing. None of the three keeps a structured memory of why an earlier grouping was rejected. The research file found no work linking learned clause retention to thematic analysis.

Our contribution is a specific loop rather than a generic "add a solver." It resembles how solvers for richer logics (satisfiability modulo theories) call a specialist checker. Here the SAT solver handles the combinatorics of grouping, and LLM critics play the specialist checker for semantic coherence. The loop rests on three pieces. Rejections are minimized before storage, so one verdict rules out a whole family of future groupings. Clauses are retained incrementally through assumption literals, the switching mechanism of incremental bounded model checking, so judgments can be added, reused, and retracted without rebuilding the problem. Finally, unsatisfiability under the current clauses is read as evidence that a new transcript needs a new theme.

The prediction that could fail is this. If plain prompt memory of rejected themes matches minimized clauses on convergence and quality, the idea reduces to bookkeeping, and the experiments are designed to detect exactly that outcome.

Research Program

Encoding. Coding agents produce codes $c \in C$, each linked to passages. Boolean variables $x_{c,t}$ indicate that code $c$ is placed in theme slot $t \in \{1,\dots,K\}$. The hard constraints are:

  • every code is covered by at least one theme, a formally checkable coverage requirement where current systems rely on LLM or human ratings;
  • the number of non-empty themes stays at most $K$;
  • symmetry-breaking constraints prevent the solver from re-proposing a relabeled copy of a grouping it has already seen.

Soft preferences come from code-pair compatibility scores and are handled with the optimal-subset techniques of OCUS. That work shows that warm-starting the solver with previously found subsets turns hours into seconds, and we use it as a method source.

Learned clauses from critics. When a critic agent rejects a theme with code set $S$, we do not store "theme $S$ was bad." Instead we run deletion-based minimization: we drop one code at a time and ask the critic whether the reduced set is still incoherent. The result is a minimal incoherent core $M \subseteq S$. We then add, guarded by a fresh switch literal $g_M$, the clause

$$g_M \rightarrow \bigwedge_{t=1}^{K} \neg \bigwedge_{c \in M} x_{c,t},$$

which says that while $g_M$ is on, the codes in $M$ may never all share any theme slot. Minimization costs $O(|S|)$ extra critic calls. The underlying idea, that querying in proportion to a small set beats querying in proportion to the whole set, follows Literal-Based MCS Extraction. Redundancy verdicts ("themes A and B overlap") are encoded the same way, as clauses over shared minimal code subsets.

Retraction and noise. LLM critics are wrong sometimes, and a permanent wrong clause would be worse than forgetting. Because every clause sits behind a switch, a human reviewer, or a majority vote over several critic personas, can disable it without discarding the rest of the solver's state. This uses assumption literals as incremental BMC and CVC3's push/pop interface do. It also directly addresses the deletion problem listed as open for incremental solving. The clause set doubles as an audit trail: a reflexive researcher can read, contest, and retract the exact judgments that shaped the final themes. This is the human role that TAMA keeps, now attached to explicit objects rather than free-text feedback.

Incremental transcripts and novelty. Transcripts are added one at a time. The new transcript's codes are asserted under temporary assumptions. If the solver can extend the current themes while keeping all retained clauses, the transcript is absorbed. If not, the unsatisfiable core names the codes and clauses in tension, and we flag a candidate novel theme. This turns the "saturation" plateau observed in Qualitative Code Suggestion into a measurable event. It also tests whether structural evidence helps novelty detection, where that paper found neural models no better than BM25.

Baselines

The naive reference point is a single long-context GPT-4o pass that reads all transcripts and outputs themes, with no refinement. It shows how much any loop buys at all.

The strongest existing method is the autonomous multi-agent refine loop of Auto-TA, with the evaluation criteria of TAMA. We reimplement it with the same base model, the same coding-stage outputs, and the same theme budget (12, matching SFT-TA). Its round limit is set so that its total LLM-call budget equals ours, including our minimization calls. For reference, TAMA reports Hit Rate rising from 83% to 92% after refinement and inter-theme Jaccard similarity of 29%, against 33% for human themes. SFT-TA reports +10.3% Fuzzy Match, +15.3% Cosine Similarity, and +22.8% METEOR over base GPT-4o on the AAOCA dataset. Beating this baseline means matching or exceeding Hit Rate against the human reference themes at no worse Jaccard redundancy. It also means achieving measurably lower regression and higher run-to-run agreement under the same compute budget.

The key ablation is the prompt-memory loop. It is identical to our system, except that each rejected theme is appended verbatim to a shared "rejected groupings" blackboard visible to every agent, with no solver and no minimization. A second ablation keeps the solver but stores unminimized clauses (the whole rejected set $S$). Together these separate three effects: having memory at all, having it enforced by a solver, and generalizing each rejection to its minimal reason.

For novelty detection, the baseline is Okapi BM25 retrieval against existing codes, which Qualitative Code Suggestion found neural models could not significantly beat on CVDQuoding. No reported numeric result for BM25 novelty detection on CVDQuoding was found on Lacuna, so we will rerun it on our splits.

Why not just…?

Why not just keep a list of rejected themes in the prompt? This is the most serious threat, and it is the main ablation. A verbatim list blocks exact repeats but not the near-variants the solver must rule out. A theme {a, b, d} is not textually the rejected {a, b, c}, yet it contains the same incoherent pair. Lists also grow until they compete for the context window. TAMA and Auto-TA already keep refinement history in prompts and still report stagnation and regression, which is circumstantial evidence against this alternative. Still, if the prompt-memory ablation matches us on the regression rate and on Hit Rate, we will conclude that the solver adds nothing and report that.

Why not just solve a one-shot set-cover or clustering problem over embeddings? It would be cheap and deterministic. However, embedding similarity confuses topical overlap with qualitative meaning. That is the reason Text Annotation via Inductive Coding finds automated methods failing on abstract text, and the reason neural retrieval does not beat BM25 for novelty. Our solver-only variant with embedding-derived soft constraints and no critic clauses tests this directly. If it matches the full system, the LLM critics are unnecessary.

Why not just fine-tune a coding model? Fine-tuning targets code quality, not the stability of the refinement loop. Its gains plateau near 100 examples (Text Annotation via Inductive Coding). SFT-TA shows that fine-tuned agents work best inside a checks-and-balances pipeline, which is the part we change. The two approaches are complementary, and we will run our loop with SFT-style coders where training pairs exist.

Proposed Experiments and Decisive Checks

The primary data is the AAOCA parent focus-group corpus used by TAMA, SFT-TA, and Auto-TA (9 sessions, 42 parents, human reference themes from 30 hours of manual work). We will request access from those authors. CVDQuoding (15 transcripts, dual-coder annotations) is used for incremental and novelty experiments because it has a natural transcript order. The single psychedelic art-therapy transcript is used for the stochastic-reliability protocol of Multi-LLM Thematic Analysis: six runs at temperature 0.7, 15 pairwise comparisons, a cosine threshold of 0.70, and Cohen's κ, where Gemini 2.5 Pro reached κ = 0.907. All systems share one base model and an equal LLM-call budget.

The first decisive check is regression. An independent held-out judge, a different model family from the critics, counts proposed themes in later rounds that contain a pair previously rejected by majority vote. Our system should drive this to zero by construction. The test is whether it does so without hurting Hit Rate, and whether the prompt-memory ablation also drives it near zero.

The second check is quality, measured by Hit Rate against human themes, Jaccard redundancy, and formally verified code coverage. Blinded expert ratings on TAMA's four criteria (Coverage, Actionability, Distinctiveness, Relevance) are collected for a subset.

The third check is dependability: κ and semantic agreement across six seeds. We expect gains here, because retained clauses push different runs toward the same feasible region.

The fourth check is incremental cost and novelty. Adding CVDQuoding transcripts one at a time, we measure LLM calls per added transcript against full recomputation. We also measure the precision and recall of the solver's unsatisfiable-core novelty flags against codes first introduced by human coders in that transcript, compared with BM25.

Finally, a retraction study tests robustness to critic noise. We inject known-wrong clauses and measure how much quality is recovered when humans or vote-based retraction disable them. This tells us whether solver memory amplifies critic errors more than prompt memory does.