When a general language model is fine-tuned on medical, legal, or chemistry text, it becomes more confident about domain text. Some of that confidence comes from new knowledge, for example that a particular drug interacts with another. Much of it may come only from the domain's surface style: its vocabulary, its stock phrases, its way of hedging. We propose to separate these two sources at the level of individual tokens. The proposal compares the specialized model's probabilities with those of the general checkpoint it came from. It then subtracts the part of that shift that a cheap word-sequence statistics model already predicts. That cheap model is an $n$-gram model, meaning a model that predicts the next word from counts of the previous few words, trained once on the domain corpus and once on general text. Our hypothesis is that the confidence gain left after removing style tracks whether the model actually learned the fact. Fabricated claims from specialized models should therefore show up as tokens whose confidence gain is almost fully "explained" by style.
This matters because specialization makes hallucinations harder to see. A clinical model that invents a dosage does so in fluent clinical prose. Fluency is exactly what likelihood-based detectors reward. Specialization also wears away general knowledge, so the same model may fabricate on general questions in a confident domain register. The method needs two forward passes and a table lookup, works on a single response, and needs no judge model. That makes it practical in the streaming, one-answer-at-a-time settings where clinical deployments operate.
Core Claim
Let $P_{\text{spec}}$ be the specialized model and $P_{\text{base}}$ its parent checkpoint. Let $Q_{\text{dom}}$ and $Q_{\text{gen}}$ be $n$-gram models trained on the adaptation corpus and on a matched-size general sample. For a generated token $w_t$ with context $w_{<t}$, we split the specialization gain as
$$ c_t = \underbrace{\big[\log P_{\text{spec}}(w_t\mid w_{<t}) - \log P_{\text{base}}(w_t\mid w_{<t})\big]}_{\text{total gain } g_t} \;-\; \lambda \underbrace{\big[\log Q_{\text{dom}}(w_t\mid w_{t-n+1:t-1}) - \log Q_{\text{gen}}(w_t\mid w_{t-n+1:t-1})\big]}_{\text{style gain } h_t}, $$
Here $c_t$ is the content gain and $\lambda$ is a single scale fitted on held-out unlabeled text by regressing $g_t$ on $h_t$. A claim's hallucination score is the negative minimum of $c_t$ over its content-bearing tokens, which are entities, numbers, and relations. The score is high when a claim contains a token the model became confident about mostly through style.
The claim is falsifiable in a specific way. The style correction must improve detection over the uncorrected contrast $g_t$. Otherwise the $n$-gram ingredient is decoration and the idea reduces to plain base-versus-specialized contrast. Gap 4 of our literature survey identifies that plain contrast as unexplored, but it is also the obvious first thing to try.
How This Relates to Prior Work
The closest work uses $n$-gram statistics for hallucination detection but stays in general domains. Measuring the Impact of Lexical Training Data Coverage on Hallucination Detection builds suffix arrays over the 1.3-trillion-token RedPajama corpus. It combines $n$-gram counts and likelihoods with model log-probabilities and raises NQ-Open detection accuracy from 64.2% to 73.3%. That result motivates us, because training-data statistics clearly carry signal. It also constrains us, because exposure in the training data is a strong baseline we must beat or absorb. We differ in what the $n$-gram model is used for. In our method it is a nuisance model of style that gets subtracted, not a measure of coverage that gets added. It is trained on the much smaller adaptation corpus, which avoids indexing trillions of tokens.
Beyond ROUGE: N-Gram Subspace Features extracts singular-value signatures from $n$-gram co-occurrence tensors. It reaches F1 0.963 on HaluEval-Dialogue, against 0.552 for perplexity. However, it needs batches of 20 outputs and is untested under specialization. Our method scores one response at a time.
On the domain side, SpikeScore shows that activation probes collapse to AUROC 0.5–0.6 when moved across domains. Its own score recovers roughly 0.78–0.79 by inducing 20 dialogue turns per answer. We aim for similar domain robustness without that cost. Our method is training-free by construction, so it has no probe to overfit to one domain. ClinicBench and MHB supply the motivation and the evaluation ground. ClinicBench shows that clinical instruction tuning helps clinical reasoning while hurting summarization. MHB shows that physician-built traps catch even strong models, with a 59.0% hallucination rate for Llama-4-Scout. Neither offers a detector.
The likelihood-evaluation literature supplies both the method and a warning. GPTScore shows that conditional log-probability is a usable training-free quality signal. It also finds that aligned checkpoints score worse than base ones. LLMs as Narcissistic Evaluators shows that models favor text close to their own distribution. Together these say that a specialized model's raw confidence is contaminated by its own register. That contamination is what our subtraction targets. Zipporah and AM-FM establish that $n$-gram likelihood measures fluency and naturalness rather than correctness. We rely on that property to use the $n$-gram model as the "style only" reference.
There is a tension in the literature. GPTScore suggests post-training degrades the value of log-probabilities, while the Eval4NLP 2023 shared task finds internal likelihood more stable than prompting on aligned models. Our design addresses this directly. The base checkpoint acts as an anchor for whatever calibration post-training has distorted.
Research Program
We build controlled specialization pairs so that the adaptation corpus is known exactly. Llama-3.1-8B, one of the models studied in SpikeScore, is continually pretrained separately on a medical corpus and on a mathematics corpus. A third variant receives clinical instruction tuning in the style of ClinicIFT. We also evaluate released pairs such as Meditron-70B and PMC-LLaMA against their LLaMA parents, wherever the adaptation data can be reconstructed closely enough to train $Q_{\text{dom}}$.
For each pair we train modified-Kneser-Ney 5-gram models, a standard smoothed count-based estimator, on the adaptation corpus and on a size-matched general sample. We then compute $g_t$, $h_t$, and $c_t$ for every generated token.
The analysis has three parts. The first is detection: whether the claim-level score flags hallucinations in both the target domain and the general domain. The second is a token-level audit. Annotators mark fabricated spans, and we test whether those spans have high $h_t$ and low $c_t$ more often than correct spans with similar total gain. The third is a test of the specialization tax. On general-domain questions, tokens where $P_{\text{spec}}$ falls below $P_{\text{base}}$ should mark displaced knowledge. The share of answer tokens with style-driven positive gain should rise as hallucination rates on general benchmarks rise, across adaptation checkpoints. If it does, the measure would give an early warning of the tax before task accuracy drops, which is Gap 3 in our survey.
Baselines
The naive reference point is the specialized model's own length-normalized log-probability of the answer. This is the GPTScore-style score without an aspect instruction. It matches what a deployer would try first, and it is the weakest result we should beat. The closest published analogue reports 64.2% accuracy on NQ-Open for a log-probability baseline (Measuring the Impact of Lexical Training Data Coverage). We also report the base model's log-probability of the same answer, which tests whether the parent alone already knows more than the child.
The key ablation is the uncorrected contrast, scored as $-\min g_t$ over the same content tokens with the same token selection. If this matches the full method, the $n$-gram correction contributes nothing and we will report it that way. A second ablation replaces the trained $n$-gram style model with a unigram frequency ratio. This tests whether local word order matters or only vocabulary does.
The strongest existing statistical competitor is the lexical-coverage classifier of Measuring the Impact of Lexical Training Data Coverage, which reports 73.3% accuracy on NQ-Open. We re-implement it in two versions, one with counts over the adaptation corpus and one with counts over the general corpus. It uses the same decision-tree and MLP classifiers trained on the same labeled split. Our content-gain features are fed into that identical classifier as well. This keeps tuning budgets identical and lets us test whether our features add information beyond coverage.
The strongest domain-robust detector is SpikeScore, which reports average cross-domain AUROC of about 0.78–0.79. We run it on the same 8B pairs with its published prompt library. We report AUROC together with tokens consumed per sample, since our method is meant to win on cost at comparable accuracy. A frontier judge prompted with a verification rubric is included as a reference point. Beyond ROUGE reports GPT-4o-as-judge at 0.748 accuracy on HaluEval-Summary.
On MHB we found no reported detection result on Lacuna, only generation hallucination rates. All MHB detection numbers will therefore come from our own runs.
The bar for success has three parts. The style-corrected score must beat the uncorrected contrast by a margin outside bootstrap confidence intervals on at least the clinical and general-domain test sets. It must add accuracy on top of the coverage classifier when combined with it. It must reach within a few AUROC points of SpikeScore in cross-domain transfer at a small fraction of its cost.
Why not just…?
Why not just use the contrast between the specialized and base model, without any $n$-gram model? This is the real threat. The base model already partly encodes style, so $g_t$ may be enough on its own. Nothing in the file shows that it fails, because Gap 4 identifies this contrast as unstudied rather than refuted. The ablation above settles the question. Our specific prediction is that the uncorrected contrast will look good on in-domain questions but degrade on general-domain questions answered in a clinical register. There, style gain and fabrication coincide.
Why not just count how often the answer's $n$-grams appear in the training data? Coverage helps, as the 64.2% to 73.3% gain on NQ-Open shows. But coverage cannot tell apart a phrase the model saw and learned from a phrase it saw only as a template. The same paper also reports that 44% of 5-grams in NQ-Open prompts never appear even in 1.3T tokens. Sparsity in a much smaller domain corpus will be worse. We test coverage as a baseline and in combination with our features rather than assuming it falls short.
Why not just train an activation probe on labeled domain hallucinations? SpikeScore reports that such probes drop to AUROC 0.5–0.6 under domain transfer. Specialized deployments rarely have labeled hallucinations in every subdomain. A probe trained on our clinical labels and tested on math, and the reverse, is included so that this failure is measured on our pairs rather than cited.
Why not just ask a strong model to check the answer? HaluEval finds ChatGPT at 58.53% accuracy recognizing summary hallucinations. MHB shows strong models falling into clinical traps themselves. Judges also cost money per claim, which VeriTrail estimates at $0.09–$0.14 per claim. The judge stays in our comparison as an accuracy reference, not as the target to replace.
Proposed Experiments
Clinical evaluation uses MHB-Dialogue and MHB-Report, which carry physician-validated labels, together with ClinicBench's treatment-recommendation and emerging-drug tasks. For the ClinicBench tasks we annotate claims in specialized-model outputs against the reference answers. General-domain evaluation uses HaluEval QA, Dialogue, and Summary, plus NQ-Open and TriviaQA with correctness labels following the coverage paper. All of these are answered by the specialized models so that we measure the specialization tax. Transfer uses the math-specialized pair tested on clinical data and the reverse. Metrics are AUROC and accuracy at a threshold tuned on a held-out split, plus forward-pass tokens per sample.
The decisive check is the gap between the full method and the uncorrected contrast. It is broken down by in-domain versus general-domain questions and by whether the fabricated span is high-style. If the gap is positive and concentrated where style gain is high, the central hypothesis is supported. If the gap is absent everywhere, the $n$-gram ingredient fails. We would then report the plain contrast as the useful finding.
A second check is the dependence on corpus size. We train $Q_{\text{dom}}$ on 1%, 10%, and 100% of the adaptation corpus. If the method works only with the full corpus, it is unusable for released models whose data is private, and we will state that limit.
References
- Measuring the Impact of Lexical Training Data Coverage on Hallucination Detection in Large Language Models
- Beyond ROUGE: N-Gram Subspace Features for LLM Hallucination Detection
- Beyond In-Domain Detection: SpikeScore for Cross-Domain Hallucination Detection
- Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark
- MHB: Medical Hallucination Benchmark for Large Language Models in Complex Clinical Tasks
- GPTScore: Evaluate as You Desire
- LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores
- Zipporah: a Fast and Scalable Data Cleaning System for Noisy Web-Crawled Parallel Corpora
- AM-FM: A Semantic Framework for Translation Quality Assessment
- The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- VeriTrail: Closed-Domain Hallucination Detection with Traceability