Did the Model Know, or Just Guess the Typical Filler? Class-Level Plausibility Priors for Multilingual Hallucination Detection

When a language model writes "she was born in Kuala Lumpur and played for Selangor FC" about a little-known Malaysian footballer, it may be recalling a fact. It may also be filling the slot with whatever a typical Malaysian footballer would plausibly have. We propose to tell these apart with one extra scoring pass. We take the generated text and rescore each claim after replacing the specific entity with its semantic class (for example, "a Malaysian footballer"). If the model assigns the claimed argument the same probability with or without knowing who the entity is, the claim came from a generic plausibility prior, not entity-specific knowledge. We treat such claims as likely hallucinations. This applies an old idea from linguistics, selectional preference: the tendency of a predicate to expect certain kinds of arguments. Here it is used to measure how much of a generation is prior-driven guessing.

The setting where this matters most is low-resource languages. Factual precision falls steadily as language resources decline, while the strongest cheap detectors were built and tested on English and a few European languages. Our hypothesis is that much of this multilingual hallucination gap is "selectional fallback": the model knows less about the entity in the target language and drifts toward typical fillers. Our score needs no target-language training data, retrieval corpus, or entailment model, so it is a candidate detector exactly where those resources are missing.

Core Claim

Let $c$ be a generated claim with argument span $a$, and let $x_e$ be the preceding text mentioning entity $e$. Let $x_{K(e)}$ be the same text with every mention of $e$ replaced by a short description of its semantic class $K(e)$. We define the entity-evidence score

$$ s(c) = \frac{1}{|a|}\left[\log p_\theta(a \mid x_e) - \log p_\theta(a \mid x_{K(e)})\right], $$

where $p_\theta$ is the generating model's own probability and $|a|$ is the number of argument tokens. The second term is a class-level selectional prior: how plausible this argument is for any member of the class. When $s(c)$ is near zero or negative, the model's choice is explained by the class prior alone. Our claim is twofold. First, $s$ carries hallucination signal that the model's raw token entropy does not. Second, that signal survives the move to low-resource and non-Latin-script languages better than detectors trained on English entailment data.

The class-level substitution is the selectional-preference ingredient, and it is what separates the score from a generic "with versus without context" contrast. Removing the entity entirely, or replacing it with "someone," gives a prior that is too weak: nearly everything then looks entity-specific. Replacing it with a different named entity gives a prior that is too noisy. Resnik's A Class-Based Approach to Lexical Discovery measured how much a predicate shifts the distribution over WordNet noun classes. We invert that move. We ask how much the specific entity shifts the model's distribution over arguments beyond what its class already predicts.

Where This Sits in Prior Work

The research file reports no prior work that uses selectional preferences as a hallucination signal for generative models. The obvious version of that idea does not work, and we want to say why up front. The hallucination survey Factual Inaccuracy and Hallucination in Large Language Models lists entity and role swapping as a major failure. An example is confusing donor and patient attributes (Léo Milecki). These look like the textbook violation "the boulder swallowed the giant." Factual hallucinations are different, though: they are almost always plausible. A donor and a patient are both acceptable subjects of "received a diagnosis." A plausibility filter, whether classical or neural, would therefore pass them.

Two lines of selectional-preference work set the scope of the proposal. The ASER papers (ASER: A Large-scale Eventuality Knowledge Graph; ASER: Towards Large-scale Commonsense Knowledge Acquisition via Higher-order Selectional Preference over Eventualities) mine plausibility over events at scale. They note two problems: reporting bias, and the difficulty of mapping instances to concepts without losing nuance. That warns us against relying on an external plausibility table. We instead read the prior out of the generating model itself, conditioned on a class description, which also sidesteps the parser dependence that ASER-style extraction has outside English. Cross-lingual Induction of Selectional Preferences with Bilingual Vector Spaces and Automatic Selectional Preference Acquisition for Latin Verbs show that class-level preferences can be estimated with little target-language data. The first maps triples into English and reports correlations up to .48 with German human judgments. The second clusters arguments from small Latin corpora. Both judge plausibility as the goal. We use plausibility only as the reference point for measuring knowledge.

Three studies motivate the cross-lingual framing. Multilingual Hallucination Gaps in Large Language Models adapts FACTSCORE (a metric that splits text into atomic facts and checks each against Wikipedia) to 19 languages over biographies. Their FACTSCORE falls from about 73.7% for very-high-resource languages to 56.9% for low-resource ones. They also find that larger models still show the gap. They diagnose the gap but do not propose a mechanism or a fix. CCL-XCoT frames the gap as knowledge present in English but not reachable in Tamil or Malay. It fixes this with contrastive continued pretraining plus English-pivot reasoning, raising Tamil hallucination-free answers on CRAFT to 55% for an 8B model, against 24% for a 70B base model. That method is a training-time intervention. Ours is a training-free diagnostic, and the two can be combined.

Two benchmarks constrain the design. MUCH shows that the best claim-level uncertainty method, adapted CCP (a method that perturbs claim tokens and scores the effect with an entailment model), costs 124% of generation compute. It also shows that entailment-based detectors degrade outside English. Any method we propose must therefore be cheap and must not rely on English entailment models. Mu-SHROOM finds that retrieval-based systems win at span-level detection across 14 languages, yet even with Wikipedia in context, systems miss many hallucinations.

Research Program

The first component is the scorer. Biographies make the class definition concrete. $K(e)$ is a nationality-plus-occupation description built from the entity's Wikidata type, written in the target language. Claims come from the MUCH segmenter for the European languages. For other scripts we segment with the generating model itself, prompting it to mark the predicate and argument span of each claim. This is the predicate-argument unit that the file's Gap 4 identifies as unexplored.

Scoring requires one teacher-forced forward pass over the edited text. This pass runs in parallel over tokens, so its cost should be a small fraction of autoregressive generation. We will report measured wall-clock overhead against MUCH's figures rather than assume it.

The second component is a diagnostic study of the multilingual gap. For each language tier in the multilingual FACTSCORE biographies, we measure two things. The first is the fraction of unsupported atomic facts that have low $s$, meaning they are prior-explained. The second is how often those unsupported facts match the class-modal filler, meaning they are the most likely answer under $x_{K(e)}$. If the selectional-fallback hypothesis holds, both rise as resources fall. This would give a mechanistic reading of the gap reported in Multilingual Hallucination Gaps in Large Language Models.

The third component is a mitigation test. The model drops or hedges claims with low $s$. We plot FACTSCORE against the number of facts retained, so that gains from simply saying less are visible.

Baselines

The naive reference points are flagging every claim and random scoring. These anchor the ROC-AUC at 0.5. On the biography mitigation test, the matching naive reference is uniform random dropping of facts at matched retention.

The cheap competitor is token entropy computed from the same model's logits. On MUCH it reaches ROC-AUC 0.737 at negligible cost (15 seconds total), as reported in MUCH. It is the strongest baseline in the same cost class, and it is language-agnostic. We will compute it on identical claim segmentations. Beating it means a higher ROC-AUC on MUCH and on our non-European splits. It also means a statistically significant gain when $s$ is added to entropy in a logistic combination, which shows the two signals are not redundant.

The strongest existing method is adapted CCP. On MUCH it reports ROC-AUC 0.772 across all languages at 124% overhead, and the best baseline there reaches only 48% true positive rate at a 10% false positive rate MUCH. We will run the released CCP configuration with the same generating model and claims. Neither method gets tuning beyond choosing a single threshold on the MUCH development split. Beating CCP means one of two outcomes. Either we match its ROC-AUC at a fraction of its overhead, or the entropy-plus-$s$ combination exceeds 48% true positive rate at 10% false positive rate. On languages outside MUCH, we expect CCP's English entailment model to be the weak point.

For span-level detection on Mu-SHROOM, the reference is the retrieval-based systems that topped the task. The file gives no per-team numbers: no reported result found on Lacuna for the top Mu-SHROOM IoU. We will therefore build a retrieval baseline ourselves, using an LLM judge with the Wikipedia context, under the same model budget. We will compare in two modes: our score alone, and our score used as an extra feature for the retrieval system.

For the mitigation test, the baseline is English-pivot prompting in the style of the XCoT stage of CCL-XCoT, run on the same base model. Prompting in English is also the condition that Multilingual Hallucination Gaps in Large Language Models found most accurate.

The key ablation isolates the class-level prior. We replace $x_{K(e)}$ with three alternatives: the entity removed, a generic "someone" or "something," and a randomly chosen different entity of the same class. If the class description does not beat these, the selectional-preference framing adds nothing and the method reduces to a generic context contrast. A second ablation writes the class description in English rather than the target language. This tests whether the prior must be native to the language, which is the tension raised between pivot prompting and the localization failures documented in GlobalWoZ.

Why not just…?

Why not just score argument plausibility directly with a selectional preference model or the LLM's own judgment? Factual hallucinations are by construction plausible. The role swaps catalogued in Factual Inaccuracy and Hallucination in Large Language Models place one acceptable filler where another belonged. We include the class-conditioned plausibility $\log p_\theta(a \mid x_{K(e)})$ alone as a detector, and we predict it performs near chance or even inverts. That result would show that the useful quantity is the gap between entity and class, not plausibility itself.

Why not just use token entropy, which is cheaper still? Entropy cannot separate "many correct answers exist" from "the model does not know." Our score targets the second case. This alternative is a real threat, because the two quantities could be strongly correlated. The experiment that settles it is the partial-correlation and combined-model test against entropy on MUCH and Mu-SHROOM. If $s$ adds no signal once entropy is controlled for, we will report a negative result.

Why not just use retrieval? Retrieval wins on Mu-SHROOM, but it depends on target-language documents, which are thin for low-resource languages and obscure entities. Even with retrieved passages, retrieval systems miss hallucinations Mu-SHROOM. We treat retrieval as complementary. If adding $s$ does not improve a retrieval system on the low-resource and surprise languages, the proposal's practical value shrinks to the no-retrieval setting, and we will say so.

Why not just prompt in English or use a larger model? Pivot prompting helps on average, but it adds latency. It can also fail to produce correct output in languages like Malay and Javanese Multilingual Hallucination Gaps in Large Language Models. It also distorts local entities GlobalWoZ. Scaling alone leaves the gap in place, as both CCL-XCoT and the hallucination-gap study report. We run the score on top of pivot generations as well. This tests whether pivoting removes selectional fallback or only moves it into the translation step.

Decisive Checks

The idea is supported if three things hold. First, class-level substitution beats all three alternative priors. Second, $s$ improves on token entropy when the two are combined. Third, the share of prior-explained unsupported facts rises monotonically down the resource tiers of the 19-language biography benchmark.

Selectional fallback would gain direct evidence as a driver of the multilingual gap if the third check holds and low-$s$ dropping also closes more of the low-resource FACTSCORE gap than entropy-based dropping at equal retention.

One known failure mode we will report separately is true facts that are typical of the class, such as a Malaysian footballer born in Malaysia. These may receive low $s$ and be falsely flagged. We will break down accuracy by how typical the true answer is under the class prior. That breakdown shows whether the method mainly penalizes legitimate typicality.