A common way to catch a language model making things up is to read its own confidence. If the model was unsure, measured by high entropy or low probability over the tokens that express a claim, the claim is flagged as a possible hallucination. These detectors are cheap and need no labels. They were built and tested almost entirely on Latin-script, general-domain text. A token's probability, though, reflects more than whether the model knows a fact. It also reflects how strange the string looks: how many subword pieces a word is cut into, whether it is capitalized, whether it is a rare drug name, and whether the script is one the tokenizer handles poorly. This proposal argues that current white-box hallucination detectors partly measure orthographic surprise rather than factual uncertainty. The confound should grow in two settings where reliable detection matters most: non-Latin scripts and specialized domains such as clinical medicine.
We propose a direct test of that confound and a cheap correction. The test re-scores the same generated claims after changing only their written form, either converting Simplified to Traditional Chinese or lowercasing German nouns, and checks whether detector scores move. The correction subtracts, from the model's surprisal on a claim, the surprisal that a small domain- and script-specific character language model assigns to the same string. The idea borrows from work showing that character n-gram models capture the "shape" of names across scripts. That small model sees no facts, only orthography. Whatever the large model finds surprising beyond it is a better estimate of what the large model does not know.
Core Claim
Token-level and claim-level uncertainty scores mix two sources: factual uncertainty, and the surface improbability of the character string under the tokenizer and script. The second source varies systematically with script and with domain vocabulary. As a result, a detector's quality, and even its ranking of claims, changes when only the orthography changes. A reference-corrected score, normalized per character rather than per token, should be more stable under orthographic edits. It should also narrow the detection gap between scripts, with the largest benefit on domain-fine-tuned compact models. The claim is falsifiable at each step. If orthographic edits barely move raw scores, or if plain character-length normalization closes the gap on its own, the reference model is unnecessary and we will say so.
Grounding in Prior Work
The motivating evidence comes from three places. MUCH released claim-level hallucination labels with stored token logits for four open models in English, French, German and Spanish. It found that Conditional Concept Probability reaches ROC-AUC 0.772 at 124% runtime overhead, that token entropy reaches 0.737 almost for free, and that the best detector reaches only 48% true positives at 10% false positives. Its rule-based claim segmenter depends on Latin punctuation and whitespace. MUCH therefore measures how well detectors work across languages, but it never holds content fixed while varying script, which is the control our test adds. ANHALTEN found that German-specific morpho-syntactic cues such as gendered articles help a trained token-level detector. That is direct evidence that detection signals are entangled with language-specific surface form. Its detector, however, is a supervised encoder rather than the generating model's own confidence. Multilingual Hallucination Gaps showed that factuality falls from about 73.7% for very-high-resource languages to 56.9% for low-resource ones. It measured how often models are wrong, not whether detectors can tell when they are wrong.
Two papers supply the method. Yu et al. showed that character language models separate entity from non-entity strings by perplexity across eight languages without capitalization, and that simple 6-gram models beat neural character models in this role. We reuse that finding in reverse. Instead of using character-shape surprise as a signal, we treat it as a nuisance to remove. One Form to Transfer Them All documented roughly threefold differences in token counts between native Indic scripts and romanized text. This is the tokenizer-fertility mechanism by which summed or averaged token surprisal becomes script-dependent.
Two papers constrain the design and set the domain regime. CMExam reports that a fine-tuned ChatGLM-6B reaches 45.3% accuracy while copying the format of clinical explanations and hallucinating intermediate factual steps more often. This "structural mimicry" is exactly the situation in which a fine-tuned model is fluent, and therefore low-entropy, everywhere, which weakens raw-confidence detectors. MedQA provides parallel medical exam material in English, Simplified Chinese (MCMLE) and Traditional Chinese (TWMLE), along with textbook corpora. Simplified and Traditional Chinese are close to a controlled script manipulation within a single domain. CCL-XCoT addresses the different problem of reducing low-resource hallucination by reasoning through English. It changes generation, whereas we change detection, so the two approaches are complementary.
Research Program
The corrected score for a claim $c$ is
$$ s(c) = \frac{1}{n_c}\left[\sum_{t \in c} -\log p_M(w_t \mid w_{<t}) \;-\; \lambda \left(-\log q_D(c)\right)\right], $$
where $p_M$ is the generating model's probability of token $w_t$ given the preceding text, $q_D(c)$ is the probability of the claim's character string under a small character 6-gram model trained on unlabeled domain text in the same script, $n_c$ is the number of characters in the claim, and $\lambda$ is a single weight tuned on held-out data. Dividing by characters rather than tokens removes the direct effect of tokenizer fertility. The subtraction removes surprise that a model with no factual knowledge would also feel. The same correction is applied to token-entropy terms, and, in a variant, to the probabilities inside CCP.
The program has four parts. The first builds the evaluation data. Llama 3.1 8B and a Qwen model generate free-text explanations for MCMLE, TWMLE and CMExam questions. Claims are segmented with a rule set over full-width Chinese punctuation, with spot checks by hand. Medically trained annotators label each claim against the MedQA Chinese textbooks and the CMExam expert explanations, and agreement is reported as in MUCH. The second part produces domain-specialized variants by LoRA fine-tuning the same models on CMExam training explanations, which reproduces the ChatGLM-6B setting at a modern scale. The third part runs the orthographic invariance test. Each labeled claim is converted between Simplified and Traditional characters, or each German claim in MUCH has its nouns lowercased, and the converted claim is re-scored under teacher forcing with identical content. The fourth part trains the character reference models per script and domain: general web, medical, Simplified, and Traditional.
Baselines
The naive reference point is raw mean token entropy as reported in MUCH, which costs nothing and gives ROC-AUC 0.737 on MUCH's languages. It is the score our correction modifies, so any gain must be measured against it. The strongest existing unsupervised method is Conditional Concept Probability as evaluated in MUCH, which reports ROC-AUC 0.772 at 124% overhead. For the Chinese medical setting, no reported result was found on Lacuna for claim-level uncertainty detection on CMExam or MedQA generations, so we will run CCP ourselves.
All methods score the same generations, segmentation and labels. They share a held-out split for their single free parameter: $\lambda$ for ours, the claim threshold for the others. On MUCH, we reproduce its numbers before adding anything.
Two ablations isolate the key ingredients. The first is character-length normalization without the reference model, which tests whether fertility alone explains the effect. The second swaps the domain reference model for a general-domain character model in the same script, which tests whether domain matching matters. The strongest supervised competitor is an ANHALTEN-style adapter detector on XLM-RoBERTa, trained with the same few hundred labeled claims per script that we use only for tuning $\lambda$.
We count the method as beating the baselines if it meets three conditions. Its ROC-AUC must exceed CCP on both Chinese scripts with confidence intervals that do not overlap. It must cut the Simplified-minus-Traditional AUC gap by at least half relative to raw entropy. It must achieve this at under 10% runtime overhead, while not falling below MUCH's reported numbers on Latin-script languages.
Why not just…?
Why not just length-normalize the entropy? This is the most serious threat. If tokenizer fertility is the whole story, dividing by characters fixes it, and the reference model adds nothing. Our first ablation tests exactly this. We expect it to fall short on domain vocabulary, because rare drug and gene names stay surprising per character even when the model knows the fact. That is the pattern Yu et al. found for entity strings in general. If the ablation matches the full method, we will report the simpler fix as the contribution.
Why not just use CCP or an NLI verifier, which already scores best? CCP costs more than doubling generation time according to MUCH, and the same paper reports that NLI-based verification degrades outside English because the verifiers are English-trained. More importantly, CCP is built from the same token probabilities, so it should inherit the orthographic confound. The invariance test will show whether CCP's scores also shift under Simplified–Traditional conversion.
Why not just translate to English and detect there? Translation rewrites the very tokens whose probabilities are being read. It is also the setting where ANHALTEN found that machine-translated training data lost to a small set of gold target-language examples. For clinical text, translation can also alter terminology. We will include translate-then-score with English entropy as an additional comparison, but we do not expect it to preserve alignment between logits and claims.
Why not just train a supervised detector per language? With enough labels, a supervised detector may well win. The question is cost: every script and domain combination needs new annotation. We compare at matched label budgets, and we report the crossover point at which supervision overtakes the unsupervised correction.
Proposed Experiments
The decisive check is the invariance test. For each detector we report the rank correlation of claim scores before and after a content-preserving script conversion, and the change in ROC-AUC. If raw entropy and CCP stay stable, with rank correlation above 0.95 and AUC changes within noise, the core hypothesis is wrong and the project reduces to a benchmark contribution. The second check compares base and domain-fine-tuned models. We predict that raw-entropy AUC drops after fine-tuning, consistent with CMExam's mimicry finding, while the corrected score drops less. The third check is cross-script portability: tune $\lambda$ on Simplified Chinese and apply it unchanged to Traditional Chinese, and from German to Spanish within MUCH. Reported metrics are ROC-AUC, true positive rate at 10% false positive rate (directly comparable to MUCH's 48%), and runtime overhead relative to generation.
References
- MUCH: A Multilingual Claim Hallucination Benchmark
- ANHALTEN: Cross-Lingual Transfer for German Token-Level Reference-Free Hallucination Detection
- Multilingual Hallucination Gaps in Large Language Models
- On the Strength of Character Language Models for Multilingual Named Entity Recognition
- One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography
- Benchmarking Large Language Models on CMExam - A comprehensive Chinese Medical Exam Dataset
- What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
- CCL-XCoT: An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generation