When a language model writes a paragraph, today's fact-checkers split it into short "atomic" claims, check each claim on its own against retrieved evidence, and average the verdicts. That procedure quietly assumes the model's mistakes are independent: whether claim 3 is fabricated says nothing about claim 7. The evidence points the other way. Models that commit to one false statement go on inventing supporting falsehoods to stay consistent. Hallucination rates also differ sharply by subject, with films and companies especially error-prone. This mirrors a problem topic modeling solved twenty years ago. Latent Dirichlet Allocation assumed topics were independent, and correlated topic models showed that learning how topics co-vary gives better predictions from partial observations. We propose to apply the same move to hallucination. We would model the joint distribution of claim-level errors inside a response with a logistic-normal prior whose covariance is structured by topic. Checking a few claims then updates our belief about every other claim, especially claims on related topics.
The practical payoff is cheaper, better-calibrated verification of long outputs. Suppose errors really are correlated along topical lines. Then a verifier can check a small, well-chosen subset of claims and infer the rest. It can also flag a response as unreliable before exhaustively checking it, and it can assign sensible uncertainty to claims whose evidence retrieval failed. If errors turn out to be independent once each claim's own verifier score is known, the proposal fails cleanly. Measuring that is the first experiment.
Core Claim
Within one model response, hallucination indicators are positively correlated, and the correlation is stronger between claims on related topics. A hierarchical model that captures this can do two things independent aggregation cannot. It can predict the veracity of unverified claims from a few verified ones, and it can produce better-calibrated response-level scores for a fixed verification budget. The new ingredient is a covariance over error propensities, not over word content. It is shaped by topic geometry learned from a reference corpus and fitted on labeled model outputs.
Relation to Prior Work
Three papers motivate the gap. Decomposition Dilemmas shows that decompose-then-verify loses context through omission, ambiguity, and over-decomposition. On BINGCHAT, re-injecting context through a reflection step raised F1 from 24.91 to 67.70. That paper diagnoses lost content dependencies between claims. It does not model dependence between claim errors, and its aggregation stays independent. How Language Model Hallucinations Can Snowball shows that an early wrong commitment produces follow-on justifications. The models themselves reject these justifications in isolation 67% (GPT-3.5), 87% (GPT-4), and 94% (LLaMA-2-70B-chat) of the time. This is direct evidence that errors within a response are coupled, though the paper studies only narrow symbolic tasks. HaluEval documents topic-dependent hallucination rates, but only as descriptive buckets. In our model those rates become the mean $\mu$, and the covariance $\Sigma$ is what we add.
Two papers supply the method. Correlated Topic Models replaced the Dirichlet with a logistic-normal prior $\mathcal{N}(\mu,\Sigma)$ and gained most when only part of a document was observed, which is exactly our partial-verification regime. Efficient Correlated Topic Modeling with Topic Embedding makes the covariance low-rank through topic embeddings. We reuse this to parameterize $\Sigma$ with far fewer parameters than labeled responses. Scalable Inference for Logistic-Normal Topic Models gives Polya-Gamma augmentation, which turns our logistic likelihood into a conditionally Gaussian one. That makes updating on newly verified claims a fast closed-form step.
The per-claim signals come from existing detectors. SAC3 provides black-box consistency scores, Semantic Entropy Probes provide single-pass uncertainty, and HaMI provides token-level white-box scores. All three score claims or sequences in isolation. We treat their outputs as features inside a joint model rather than competing with them. The closest topic-and-hallucination work, Addressing Topic Granularity and Hallucination in LLMs for Topic Modelling, fixes hallucinated topic labels when an LLM is used as a topic extractor. It does not use topic structure to reason about factual errors in general generation.
Model
Each response $r$ is split into claims $c_1,\dots,c_m$. A correlated topic model fitted on a reference corpus maps each claim to a topic mixture $\theta_i$ over $K$ topics. Each response gets a latent vector $\eta_r \in \mathbb{R}^K$ of per-topic error log-odds. Each claim has a feature vector $x_i$ holding its own detector scores. The probability that claim $i$ is hallucinated is
$$\eta_r \sim \mathcal{N}(\mu, \Sigma), \qquad \Sigma = UU^\top + \mathrm{diag}(d), \qquad P(h_i = 1 \mid \eta_r) = \sigma\!\left(\theta_i^\top \eta_r + w^\top x_i\right).$$
Here $h_i$ indicates hallucination, $\sigma$ is the logistic function, and $U \in \mathbb{R}^{K\times q}$ is initialized from the topic embeddings and fine-tuned. $\mu$, $U$, $d$, and $w$ are fitted on responses with claim-level labels.
At test time, a verifier returns labels for a subset of claims. The Gaussian posterior over $\eta_r$ then yields predictions for the remaining claims and for the response as a whole. Choosing the next claim to verify is a variance-reduction problem on that posterior.
The model nests two important special cases. Setting $\Sigma=0$ recovers independent aggregation. Setting $K=1$ with a scalar shared effect gives an "exchangeable" model in which every claim in a response shares one reliability level regardless of topic.
Research Program
The first stage measures the phenomenon. We take claim-labeled long-form data: FELM and BINGCHAT from the decomposition study, plus the human-annotated Alpaca portion of HaluEval, where about 19.5% of ChatGPT responses contained a hallucination. We add a constructed set: multi-question sessions about a single entity drawn from HotpotQA and the Senator Search task, answered by open models (LLaMA-family and Mistral-7B) whose hidden states we can probe. On all of these we estimate the partial correlation of claim errors given each claim's own detector score. We break the estimate down by the topical distance between claims, measured as inner products of their topic mixtures under the fitted embedding geometry.
The second stage fits the hierarchical model and its nested special cases. The third stage evaluates budgeted verification and calibration.
The topic model is trained once on Wikipedia-scale text using the embedding approach, so topics are fixed before any hallucination labels are seen. This avoids fitting topics to the error labels themselves.
Baselines
The naive reference point is FactScore-style independent verification. Every claim is checked by the same retrieval-plus-verifier pipeline, and the response score is the fraction of supported claims. Under a budget, it checks a random subset of $k$ claims and extrapolates. All systems share this exact verifier, retrieval index, and claim decomposition, so the only thing that varies is how verdicts are combined and which claims are checked.
The strongest existing detector is SAC3-all. It reports AUROC of 87.0% on HotpotQA-halu and 77.2% on NQ-open-halu (SAC3). We run it per claim with the same verifier model and count its extra perturbation queries against the verification budget. Per claim, we also run HaMI from its released code (mala-lab/HaMI). HaMI reports up to an 11.5% AUROC gain over prior probes, but the file gives no absolute value that transfers to our setting. We also run Semantic Entropy Probes. These detectors feed $x_i$ in our model, so the fair comparison is each detector alone versus the same detector inside the joint model.
For context-aware verification, we implement the reflection step from Decomposition Dilemmas, which reached F1 67.70 on BINGCHAT.
The key ablation is the exchangeable one-factor model, fitted with the same data and hyperparameter search. The full model beats the baselines if, at matched verifier calls, it improves held-out claim AUROC on unverified claims and response-level Brier score. It beats the exchangeable ablation only if topic structure adds a significant gain beyond the shared intercept. The target is a response-level accuracy equal to exhaustive verification at no more than half the verifier calls.
Why not just…?
Why not just verify every claim independently and average? When verification is free and perfect, averaging is enough, and our model would add little. But retrieval fails on some claims, and the reflection result shows verifiers are highly sensitive to context noise. Budget curves will show whether correlated inference recovers exhaustive-verification accuracy with fewer calls. If the curves coincide, the idea has no practical value.
Why not just give the verifier the full response as context (reflection)? Reflection repairs lost content dependencies, and it is a strong competitor on F1. It is still an independent per-claim decision, though, and it says nothing about which claims to check first or what to believe about unchecked ones. We combine the two. Reflection-verified claims serve as observations, and we test whether the joint model still adds calibration on top.
Why not just use one reliability score per response, such as a semantic entropy probe on the whole generation? This is the real threat. Snowballing may create a response-wide error mode that has nothing to do with topics, in which case a single shared factor captures everything. The exchangeable ablation tests this directly. If it matches the topic-structured model, we will report that topic covariance is unnecessary and the contribution shrinks to hierarchical aggregation. The topic version is justified only if error correlation measurably decays with topical distance in the first-stage analysis.
Why not just train a neural classifier over claim neighbors? A graph network could learn couplings, but labeled long-form data is small: FELM and BINGCHAT are modest in size. A low-rank prior tied to pre-trained topic embeddings is the data-efficient choice. We include a small graph-attention baseline on identical features to test whether the extra flexibility pays off.
Proposed Experiments and Decisive Checks
The first decisive check is whether claim errors are correlated at all once own-claim scores are known. Using held-out responses, we test whether the partial correlation is positive and whether it decreases with topic distance. A near-zero result ends the project.
The second check is predictive. We hold out 50% of claims per response, reveal the rest, and compare AUROC on the hidden half across the independent, exchangeable, and topic-covariance models. We do this on FELM, BINGCHAT, and the constructed entity sessions, reporting per-dataset results and not pooled ones.
The third check is the budget curve: response-level accuracy and Brier score as a function of verifier calls, comparing random selection against variance-reduction selection under each model. In a transfer version, the model is fitted on general-domain sessions and applied to BioASQ-style biomedical questions, testing whether topic-structured covariance transfers the way HaMI's detector does.
As a sanity check on the mechanism, we verify on the Senator Search sessions that the fitted $\Sigma$ couples claims downstream of a first wrong commitment. If it does not, the correlation we measure is not the snowball effect the model was motivated by.
References
- Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance?
- How Language Model Hallucinations Can Snowball
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- Correlated Topic Models
- Efficient Correlated Topic Modeling with Topic Embedding
- Scalable Inference for Logistic-Normal Topic Models
- SAC³: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency
- Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs
- Robust Hallucination Detection in LLMs via Adaptive Token Selection
- Addressing Topic Granularity and Hallucination in Large Language Models for Topic Modelling