We cannot currently tell whether a hallucination detector works in Bengali, because we cannot cheaply tell whether a Bengali sentence is true. Multilingual hallucination benchmarks get their "truth" labels from a large model acting as judge, or from matching claims against Wikipedia. Both break outside English. Judges pick a winner between two equally wrong answers far more often than native speakers do, and low-resource Wikipedias are too sparse to confirm true facts. The benchmarks are also static web text that future models will train on.
This note proposes building the truth instead of looking it up. We procedurally generate a small fictional world, stored as a database of people, organizations, places, dates and relations. We render it as source documents in many languages, ask a model to answer or write about that world, and label every claim in its output by running a database query against the world. In this setting a query's result is the ground truth, the way a test suite is ground truth for generated code. Every release uses a fresh random seed, so memorizing a past release does not help. The result is a multilingual hallucination benchmark whose labels involve no model's judgment of truth, cover any language we can render into, and stay valid as models are retrained.
Core Claim
The central claim is that execution-grounded labels change the conclusions we draw about multilingual hallucination detectors. Measured on such labels, two things will look different from what silver-label benchmarks report. First, the ranking of detection methods will differ, and the reported drop from English to other languages will turn out to be partly an artifact of noisy labels. Second, a specific failure mode will become visible and measurable for the first time: "prior-override" hallucination, where a model replaces a fact stated in its context with the version it memorized in pretraining. We expect this mode to grow as resource level falls.
The second point is what ties hallucination to contamination. Half of the generated worlds are edited copies of real entities, for example a real company given a different founding year. A model that asserts the real year despite the context is hallucinating because of memorized training data. We can count exactly how often that happens, per language.
The scope needs to be stated plainly. Because the facts live in a supplied world, this benchmark measures hallucinations that contradict the provided context (the "context-conflicting" category in the taxonomy of Siren's Song in the AI Ocean), plus prior-override errors in the edited worlds. It does not measure open-world factual recall. That is a deliberate trade: we give up open-world coverage to get labels that are exact in every language.
Relation to Prior Work
Two lines of work motivate the gap. Multilingual factuality benchmarks get their labels from text-based judgment:
- MUCH releases claim-level uncertainty scores for four Latin-script languages. Its labels are a silver standard: GPT-4o and GPT-4.1 cross-check claims against Wikipedia.
- Multi-FAct and An Analysis of Multilingual FActScore extend atomic-fact scoring to around eleven languages. The second paper documents both the "Wikipedia ceiling" (Bengali scoring accuracy is 60.6% without web search) and the "translation trap" (translating to English first makes judges too generous).
- PsiloQA builds span-level training data across 14 languages using GPT-4o annotation.
PARIKSHA supplies the reason to distrust those labels. Across Indic languages, GPT-4 judges agree poorly with native raters on direct assessment, with a Fleiss kappa of about 0.31.
Three papers come closest on method, and each stops short of this proposal:
- PROGRAMFC turns English claims into small verification programs. Its program steps, however, call a FLAN-T5 question-answering model rather than a ground-truth store, so the "execution" still ends in a model's opinion.
- MASSIVE-AMR maps utterances in more than 50 languages to SPARQL queries (a database query language) and executes them, reaching over 90% executability. It runs against the static DBpedia knowledge graph, though, which returned records for only about 32% of its queries. It also studies parsing hallucinations, not hallucinations in model generations.
- DyVal generates fresh reasoning problems from random graphs to defeat memorization. DyCodeEval shows that fine-tuning on leaked seeds inflates static scores (about 20% to 80%) while scores on regenerated variants stay flat (about 13%). Both are English-only and concern logic or code, not factual claims.
TCG-Bench is the one contamination-resistant benchmark here that is bilingual (English and Arabic). It measures game strategy, not factuality.
This proposal reuses DyVal's procedural generation and DyCodeEval's leak-and-retest protocol. It borrows the execution-as-oracle principle from code evaluation, as in CodeHalu, which labels code hallucinations by running the code. It applies all three to multilingual factual text.
Research Program
The world generator. The generator samples a typed knowledge graph of a few hundred entities and stores it in SQLite. It uses a fixed set of about forty relations, such as founded-in, employs, located-in, born-on and owns-share. Entity names are synthesized separately for each script, using phonotactically plausible syllables for Latin, Arabic, Devanagari and Bengali scripts. This gives us the full set of surface forms for every entity in every language.
Counterfactual worlds. Each counterfactual world copies a slice of real, well-known entities and edits a controlled fraction of their facts. We record which facts were edited.
Rendering into languages. Facts are first expressed through templates written by native speakers, then paraphrased by a model for fluency. A paraphrase is accepted only if a structured extraction pass recovers exactly the intended facts and nothing else; this round-trip check is itself executed against the database. Native speakers then audit a sample of the rendered documents in each language.
Tasks. Models see a subset of the rendered documents and perform two kinds of task:
- Short-form questions. These include multi-hop questions, counting questions and comparisons, each paired with a gold query whose executed result is the answer. Execution earns its place here: answers such as "how many subsidiaries are in cities founded before 1900" appear in no single document.
- Long-form generation. Models write profiles of entities.
Labeling. For short-form answers, labels are deterministic. We match the answer against the executed result using the known lexicon of surface forms.
For long-form text, claims are segmented with MUCH's rule-based segmenter, extended to new scripts. A parser model then maps each claim into the fixed relation set, and the resulting query is executed. A claim is labeled supported, contradicted, or unverifiable. Unverifiable means the entities exist but the world contains no such fact, so the claim is a fabrication.
The parser is the one learned component on the labeling path. It translates claims into queries but never decides whether they are true. We measure its error in each language against native-speaker audits and report labels with that error rate attached.
Release procedure. Each release is a new random seed. Item difficulty is fixed by world size, hop count and the edit fraction, so releases remain comparable over time, which crawling post-cutoff data cannot guarantee.
Languages and models. Languages are English, French, Spanish and German, chosen for overlap with MUCH, plus Arabic, Hindi, Bengali and Swedish. Generator models are the four open models used in MUCH (Llama 3.1 8B, Llama 3.2 3B, Ministral 8B, Gemma 3 4B), so that white-box uncertainty methods can be compared directly.
Baselines
The baselines fall into two groups: baselines for labeling, which test whether our oracle is needed, and baselines for detection, which test whether the new labels change conclusions.
The naive labeling reference is the translate-then-check pipeline. The output is machine-translated to English and judged by an English model against English documents. This is the "do nothing new" option practitioners actually use. It runs on the same claims, and our labels beat it if they agree better with native raters in each language.
The strongest existing labeling method is the MUCH-style silver standard. Two frontier judges cross-check each claim, and the claim is kept when they agree. MUCH reports Cohen's kappa of about 0.82 between its silver annotators MUCH. That figure is agreement between two judges, not agreement with truth.
We run this baseline in its strongest possible form for our setting. The judge receives the full rendered world documents in the target language, not Wikipedia, with an equal prompt-tuning budget on a held-out development world. This is the key ablation. It holds the synthetic world fixed and removes only the execution step. If execution matters, our labels should match native-rater labels better than this judge does, especially on aggregation questions, on unverifiable fabrications, and in Bengali, Hindi and Arabic. If the judge matches us everywhere, execution is not the contribution and we will say so.
Detection baselines. We evaluate the methods MUCH evaluated, at equal compute: claim-conditioned probability (CCP), reported at ROC-AUC 0.772 with 124% overhead, and token entropy, reported at ROC-AUC 0.737 MUCH. We add a multilingual span tagger trained on PsiloQA, for which no reported result found on Lacuna gives a benchmark score, and the SpanBERT approach of FunghiFunghi. FunghiFunghi reports span IoU of 0.4156 in Swedish, but ranking correlation of −0.2116 in Italian.
Here, "beating" means two things. First, a detector ranking under oracle labels that differs measurably from the ranking under silver labels on matched claims, with Kendall tau clearly below 1. Second, a non-English performance drop that shrinks or grows by more than its bootstrap interval once label noise is removed. A further target is to improve on MUCH's operating point, where the best method reached only 48% true-positive rate at 10% false-positive rate.
Why not just…?
Why not just give a frontier judge the source documents? This is the real threat, and the ablation above exists to settle it. PARIKSHA's evidence of judge failure comes from judging without references, so it does not by itself rule out a judge that has the references. Two pieces of evidence suggest the judge will still fall short in specific places. In MASSIVE-AMR, GPT-4 caught only about 24% of relation hallucinations even with an oracle list of allowed relations. And aggregation answers require computation that no document states. We expect the judge to be adequate on single-hop English claims and to fail on counts, comparisons, fabrications and low-resource scripts. If it does not fail there, the synthetic-world benchmark still stands, but the claim about execution would be dropped.
Why not just translate to English and use mature English detectors? An Analysis of Multilingual FActScore found that this setting makes judges overestimate factuality. Translation also misaligns span offsets, which span-level evaluation needs. We include translate-then-check as the naive baseline so this is measured rather than asserted.
Why not just crawl post-cutoff articles in each language? Fresh native web text in Bengali or Swahili is thin, which reproduces the Wikipedia ceiling instead of escaping it. As DyCodeEval notes, crawled items also drift in difficulty between releases. Synthetic worlds fix difficulty by construction. They cannot test open-world recall, and we accept that limit.
Why not just reuse MUCH and relabel a subset with humans? That is feasible for four European languages and a few thousand claims. PARIKSHA shows what native-rater annotation at scale costs (90,000 ratings for 200 prompts). We use human labels as the yardstick for a sample, not as the method.
Proposed Experiments and Decisive Checks
Label validity. For roughly 300 claims per language, native speakers label claims against the rendered world. We report agreement between native raters and each of three labelers: the oracle, the judge-with-documents, and translate-then-check. Agreement is broken down by question type and language. The decisive outcome is oracle agreement that holds steady across scripts while judge agreement falls.
Detector re-ranking. CCP, token entropy and span taggers are scored on oracle labels and on silver labels for identical claims. We report ROC-AUC, TPR at 10% FPR, Spearman correlation of confidence with error, and Kendall tau between the two rankings. The decisive outcome is whether the non-English drop documented in MUCH survives once label noise is removed.
Prior override. In counterfactual worlds, we measure the rate at which models assert the memorized real fact over the edited one, by language and by how frequent the entity is in the real world. We also test whether white-box detectors catch these errors any better than they catch ordinary fabrications. The hypothesis is that they catch them worse, because the model is confident in its memorized fact.
Contamination robustness. Following DyCodeEval's protocol, we fine-tune one open model on a full release, then evaluate it on that release and on a fresh-seed release. The benchmark passes if hallucination rates on the fresh seed stay at baseline while the leaked release improves.
Parser error budget. We report claim-to-query parsing accuracy per language against the audit. We also show that detector conclusions are stable when claims with low parser confidence are excluded. This separates the one learned step on the labeling path from the findings it supports.
References
- Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
- MUCH: A Multilingual Claim Hallucination Benchmark
- Multi-FAct: Assessing Factuality of Multilingual LLMs using FActScore
- An Analysis of Multilingual FActScore
- When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQA
- PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data
- Fact-Checking Complex Claims with Program-Guided Reasoning
- MASSIVE Multilingual Abstract Meaning Representation: A Dataset and Baselines for Hallucination Detection
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
- Cards Against Contamination: TCG-Bench for Difficulty-Scalable Multilingual LLM Reasoning
- CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
- FunghiFunghi at SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes