Hallucination Detectors as Telescopes: Calibrated Fusion of Span Signals Across Languages

When a language model writes a fluent answer in Basque or Farsi, a few words in it may be invented, such as a wrong date or a made-up name. Several kinds of detector can point at those words. They include a fine-tuned span tagger, a simple classifier trained on the model's hidden activations (a "probe"), an anomaly score computed on the model's output probabilities, and a large model that checks the answer against retrieved documents. Each detector sees the error differently. Some blur where the error starts and ends. Some are systematically over- or under-confident in particular languages, and some cannot run at all when the model is a closed API. This proposal treats these detectors the way astronomers treat telescopes observing at different wavelengths. Each one is an instrument with its own blur (its "point spread function") and its own calibration offset. The detectors are combined by fitting one shared underlying picture, here the per-character probability that the text is hallucinated, which every instrument observes through its own distortions.

The opportunity comes from a specific weakness. On the multilingual Mu-SHROOM benchmark, systems can localize errors moderately well but rank them badly against human uncertainty. A SpanBERT system reached intersection-over-union of 0.2943 in English and 0.4156 in Swedish, yet produced a Spearman correlation of −0.2116 with annotator probabilities in Italian (FunghiFunghi at SemEval-2025 Task 3). Astronomy has decades of practice with this exact situation: combining instruments with known resolution, cross-calibrating them where their coverage overlaps, measuring how often a match is a chance coincidence, and continuing when some bands are missing. Our claim is that a small, explicit measurement model of this kind gives better-calibrated multilingual span detection than averaging or learned stacking. The advantage should be largest in languages with almost no labels, because the model's few parameters per detector and language can be fit from agreement between detectors instead of from annotations.

What the directions contribute

Span-level detection supplies the instruments and the task. Detection has moved from whole-response judgments to marking exact character offsets (FALTE), because modern errors are partial and embedded in correct text (Xu et al. on NMT introspection). The strongest single detectors on English retrieval-grounded data are trained models. On RAGTruth, a fine-tuned Llama-2-13B reaches span F1 of 52.7 against 28.3 for prompted GPT-4. RL4HS reaches 55.9 Span-F1 with reinforcement learning over reasoning traces, compared with 50.1 for supervised fine-tuning and 51.2 for o3. These works motivate the proposal: they show that individual detectors are strong but imperfect. They do not ask how to combine detectors or how to calibrate them per language.

Multilingual detection supplies the regime and the main constraint. The Mu-SHROOM overview covers 14 languages, four of which are surprise test-only languages (Catalan, Czech, Basque, Farsi). It scores systems with both IoU and Spearman correlation against the proportion of annotators who flagged each character, and it reports large disagreement among annotators about span boundaries. Four detector families exist, and we reuse each as an instrument without changing its method:

  1. Linear probes on Llama-3.2 activations, from Deloitte (Drocks), which ranked top-10 in 13 of 14 languages.
  2. Treating the model's output probabilities over time as a sequence and scoring anomalies in it (Aryal et al.).
  3. SpanBERT start/end tagging (FunghiFunghi).
  4. Retrieval-grounded LLM verification that outputs delimiter-tagged text instead of numeric offsets, following CCNU and FENJI.

Two findings in this literature constrain the design. Probes need white-box access to the model, so some instruments are unavailable for closed models. Herrlein et al. find that zero-shot transfer from English misses target-language shifts while a few target-language examples help a lot. Together these suggest that per-language adaptation should be cheap and need very few labels.

Multi-wavelength integration supplies the method. Broadband astronomical fits do not average fluxes from different instruments. They model each instrument's response and use overlapping energy ranges as cross-calibration anchors. The joint MAGIC and Fermi-LAT Crab spectrum is an example (Aleksić et al.), and that work also shows that systematic errors such as a 15–17% energy-scale uncertainty dominate statistical ones. We borrow two specific ideas. The first is per-instrument response modeling with overlap anchors. The second is fitting with whatever subset of instruments is available. That second point is the explicit open limitation of the trimodal contrastive model in Self-supervised Multimodal Model for Astronomy, which requires complete triplets of inputs.

Unidentified high-energy sources supply an evaluation lens. Error ellipses that contain many candidates lead astronomers to quantify chance-coincidence association rates and to report an honest "unassociated" fraction. In the Fermi sky, 1FLAT reduced the high-latitude unassociated fraction from 25% to 17%, and Rangelov et al. describe the ambiguity of classification inside crowded ellipses. We import the shift-and-rematch test for spurious matches and the practice of reporting "dark" hallucinations, meaning gold spans that no instrument detects. This direction does not provide a dataset. We do not claim to work on astrophysics text.

Core Claim

Each detector $d$ produces a per-character score $s_d(i)$ on an answer in language $\ell$. We model it as a blurred, rescaled, noisy view of a latent per-character hallucination probability $p(i)$. The fused estimate is

$$\hat p = \arg\max_{p\in[0,1]^n} \sum_{d\in D_x}\sum_{i} \log \mathcal{N}\!\big(s_d(i);\ a_{d\ell}\,(k_{d\ell} * p)(i) + b_{d\ell},\ \sigma_{d\ell}^2\big) \;-\; \lambda\,\mathrm{TV}(p).$$

The symbols are defined as follows:

  • $D_x$ is the set of detectors available for this answer, so missing instruments simply drop out of the sum.
  • $k_{d\ell}$ is a one-parameter Gaussian kernel describing how much detector $d$ smears span boundaries in language $\ell$.
  • $a_{d\ell}$ and $b_{d\ell}$ are a gain and an offset, and $\sigma_{d\ell}$ is the detector's noise level.
  • $\mathrm{TV}$ is total variation, a penalty favoring contiguous spans, weighted by $\lambda$.

There are four parameters per detector and language. Where gold soft labels exist, they are fit by maximum likelihood. In the surprise languages they are fit without labels. In that case $p$ and the instrument parameters are estimated jointly by alternating updates, anchored by fixing the gain and offset of one reference instrument. We use the retrieval verifier, whose output is in text form and should transfer best across languages. This is the analog of tying satellite and ground-telescope spectra together where their energy ranges overlap.

The novel claim has three parts:

  1. The blur kernel recovers boundary uncertainty that matches human annotator spread.
  2. Per-language calibration fixes the negative rank correlations seen in single detectors.
  3. Label-free cross-calibration in unseen languages matches or beats stacking models trained on labeled languages.

Research Program

We first build the four instruments on the Mu-SHROOM training and validation data with the published recipes, using one shared base model family so that probes and logit features are comparable. Next we fit the measurement model in three regimes: full labels in the ten main languages, few-shot (20–50 answers) in held-out main languages, and label-free in the four surprise languages.

Two analyses accompany the fits. The first checks whether the fitted kernel widths $k_{d\ell}$ correlate, per language, with annotator boundary disagreement. If they do, the model is learning something real about boundary ambiguity, the open problem the overview names, rather than just smoothing. The second is the astronomy-style audit. We cyclically shift predicted spans to random positions within the same answer and recompute IoU, which gives a chance-overlap floor per language. We report excess IoU above that floor. We also report the "dark" fraction of gold characters with fused $\hat p<0.2$ and break these down by error type (entity, number, relation). This tells us which hallucinations no instrument sees.

Baselines

The naive reference point flags nothing and also flags everything. We additionally compute the cyclic-shift chance floor described above, because high IoU in languages with long gold spans can be partly coincidental. Any method must clear the chance floor before its IoU means anything.

The strongest single instruments are the Deloitte-style activation probe with retrieval and the SpanBERT tagger, each run with its published recipe and the same training data as our fusion. Neither paper reports a winning numeric IoU in the file. We have FunghiFunghi's English IoU of 0.2943, Swedish IoU of 0.4156, and Italian Spearman of −0.2116, and we found no reported result on Lacuna for Deloitte's per-language IoU, only its ranks. We will therefore reproduce both systems and also report the official top Mu-SHROOM scores from the leaderboard released with the overview. For the English retrieval-grounded setting on RAGTruth, the reference is RL4HS-7B at 55.9 Span-F1 (Learning to Reason for Hallucination Span Detection). There, fusion should at least match the best single instrument.

The strongest combination competitors are two. The first is a uniform average of detector scores after per-detector temperature scaling. The second is a learned stacking model: a small per-character classifier over all detector scores plus language identity, trained with soft annotator labels. Both receive the same labeled data, the same detector outputs, and a hyperparameter search of equal size to ours, which is 50 trials. For missing detectors, stacking gets mean imputation plus a missingness indicator, which is its strongest standard treatment.

Three ablations isolate the key ingredients. Replacing the kernel with no blur tests the boundary model. Sharing gain and offset across languages tests per-language calibration. Fitting surprise languages with English parameters instead of label-free cross-calibration tests the anchoring idea.

Fusion beats stacking if it improves mean Spearman correlation on the four surprise languages by at least 0.05 with a bootstrap 95% interval excluding zero, and does not lose IoU. On labeled main languages we expect parity. If stacking is equal everywhere, including the surprise languages, the core claim fails.

Why not just…?

Why not just train the best detector with a soft-label loss? Regressing annotator proportions directly could fix the calibration failure seen in FunghiFunghi without any fusion. This is a real threat in the labeled languages, and we include it as a baseline with an identical budget. Its weakness should appear in the surprise languages, where no soft labels exist and any calibration it learned is English- or main-language-specific. The finding from Herrlein et al. that target-language shifts escape zero-shot transfer predicts exactly this. If it matches fusion on Catalan, Czech, Basque, and Farsi, we drop the fusion claim and report that single-detector soft training suffices.

Why not just stack the detectors with a learned classifier? Stacking is more flexible than our four-parameter model and will likely win when labels are plentiful. Its cost is that it needs labeled data in each language to learn language-specific trust, and it handles missing instruments only through imputation. The decisive experiments are the surprise-language and white-box-removed conditions. If stacking still wins there, the measurement model is unnecessary.

Why not just use retrieval-augmented verification, since the overview says retrieval systems won? The Mu-SHROOM overview reports that retrieval-based submissions outperformed others, and Deloitte found that adding web search improved 12 of 14 languages. Retrieval fails silently, though, when evidence is not found, and in that case it flags correct text. It also adds latency. Retrieval is included as one instrument and as the calibration anchor. We test it alone, and we report how fusion behaves on answers where retrieval returned low-relevance passages. That subset is where combining with model-internal signals should matter most.

Proposed Experiments

The primary benchmark is Mu-SHROOM, scored with character-level IoU, Spearman correlation, and excess IoU over the chance-shift floor, reported per language. The secondary benchmark is RAGTruth, scored with span F1, to confirm that fusion does not hurt in English retrieval-grounded generation.

The first decisive check is the surprise-language comparison of label-free fusion against stacking and soft-trained single detectors. The second is a closed-model simulation in which the probe and logit instruments are removed for 50% of answers at test time. Fusion should degrade gracefully, while imputed stacking should degrade more. The third is the kernel validity check: fitted blur widths should correlate positively with per-language annotator boundary disagreement, using rank correlation across the 14 languages. A null result would mean the kernel is only a smoother.

Finally, the dark-hallucination census reports which error types no instrument recovers. This gives the field the analog of an unassociated-source fraction, a concrete target for the next detector instead of another point of aggregate IoU.

References

  1. FunghiFunghi at SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes
  2. FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation
  3. Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection
  4. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
  5. Learning to Reason for Hallucination Span Detection
  6. SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes
  7. Deloitte (Drocks) at SemEval-2025 Task 3: Fine-Grained Multi-lingual Hallucination Detection Using Internal LLM Weights
  8. Howard University - AI4PC at SemEval-2025 Task 3: Logit-based Supervised Token Classification for Multilingual Hallucination Span Identification Using XGBOD
  9. CCNU at SemEval-2025 Task 3: Leveraging Internal and External Knowledge of Large Language Models for Multilingual Hallucination Annotation
  10. FENJI at SemEval-2025 Task 3: Retrieval-Augmented Generation and Hallucination Span Detection
  11. ANHALTEN: Cross-Lingual Transfer for German Token-Level Reference-Free Hallucination Detection
  12. Measurement of the Crab Nebula spectrum over three decades in energy with the MAGIC telescopes
  13. Self-supervised Multimodal Model for Astronomy
  14. 1FLAT: A <tt>Firmamento</tt> -based Catalog of Active Galactic Nuclei in Fermi-LAT High-Galactic-latitude γ -Ray Sources
  15. Chandra X-Ray Observatory Observations of 13 Fermi LAT Sources