When someone asks a database a question in natural language, a system first has to decide whether the database can answer it at all. If it cannot, a language model will often write a plausible query anyway and return a confident wrong answer. One recent fix has a language model write plain-language descriptions of the database's tables and columns, then train a classifier to judge whether the question matches anything described. This works well in English clinical data. Nobody has tested it when the question is in Bengali, Norwegian, Arabic or Thai. In those languages, models make up facts more often, and step-by-step explanations are known to sometimes hurt rather than help.
This note proposes a specific fix. The classifier should explain its decision only in claims that the database itself can check, such as "this phrase refers to column lab.itemid", "the value 'HbA1c' occurs in that column", or "no column stores insurance status". A small program verifies each claim by schema lookup, type checking and value queries before the label is accepted. Rationales that cite nonexistent columns or impossible values are caught mechanically, not trusted. We also test whether language-specific pre-trained models, which read the local-language question better, make fewer of these linking errors than larger multilingual models.
Core Claim
Explanations help classification only when their factual content is either correct or detectably wrong. Free-form explanations in lower-resource languages are neither, which is a plausible reason why chain-of-thought prompting (writing out reasoning before answering) degraded hallucination detection in 12 of 21 model–task settings in BenHalluEval. Schema-grounded tasks are unusual because almost every factual claim a rationale needs can be checked against the database.
We claim that a classifier forced to produce schema-citing rationales, with each citation verified, will beat three alternatives at detecting unanswerable questions posed in non-English languages. The first is a description-based discriminative classifier. The second is free chain-of-thought. The third is translating the question to English before classifying. We further claim that the gap will widen as the language's resource level drops. The fraction of failed citations should also serve as a calibrated signal of misclassification.
Where This Sits in Prior Work
Our starting point is Leveraging LLM-Generated Schema Descriptions for Unanswerable Question Detection in Clinical Data, which we call SQD. It replaces raw table definitions with GPT-3.5-generated descriptions of the tables and columns in an electronic health record (EHR) database. It then scores how relevant each schema element is to the question with a Transformer encoder and a small prediction network. On EHRSQL-2024 it reaches F1 0.8547, and removing the schema tokens drops F1 to 0.5739.
SQD motivates our work in two ways. It shows that descriptions matter, and it names its own blind spot: questions whose answerability depends on what values actually exist in the database. SQD is English-only and produces no explanation. We reuse its description generator and use its classifier as our strongest baseline.
Two guardrail papers supply the method of reasoning before labeling. DynaGuard classifies conversations against user-written policies after reasoning inside <think> tags, reaching 73.1% F1 on DynaBench against 13.1% for LlamaGuard3. MR. Guard trains multilingual safety reasoning with a translation-then-cultural-variant curriculum. Both evaluate whether the rationale agrees with the label; MR. Guard reports this "fidelity" at 80–93%. Neither checks whether the facts stated in the rationale are true. We keep their reason-then-label structure but make the rationale's content externally verifiable.
Two results constrain the design. BenHalluEval shows that unconstrained chain-of-thought is unreliable in Bengali, so we cannot assume reasoning helps. MASSIVE-AMR shows GPT-4 catches only about 24% of hallucinated knowledge-base relations even when handed the list of allowed relations, so we cannot ask the model to police itself. This negative result is the direct motivation for checking citations with a program rather than a prompt. Multilingual Hallucination Gaps supplies the expectation that errors grow with lower resource levels: its FActScore (the share of generated atomic facts supported by Wikipedia) falls from about 73.7% to 56.9% under the (en, en) setup.
For the language-specific factor, the work of NLEBench+NorGLM, SeaLLMs, ArabicMMLU, and TigerLLM-9B in BenHalluEval reports that smaller models trained mainly on one language rival much larger multilingual ones. None of it touches structured data. Columbo shows that language models can expand cryptic column names and justify each expansion. We borrow its idea that descriptions need justification, and add verification of the descriptions themselves.
Research Program
The system has three parts. The first is a description writer. Following SQD, a language model writes a description of each table and column. Unlike SQD, each description is audited against the column's declared type and a sample of its distinct values, in the spirit of the typed, value-sampled prompts in ITUNLP and AlphaPro. A description that says "admission date" for a column of integer codes is flagged and regenerated. We record how often descriptions fail this audit, broken down by the language they are written in and by which model wrote them. This measures a hallucination source that SQD never examines.
The second part is a rationale-producing classifier. Given a question in language L and the audited descriptions, the model emits a short structured rationale. The rationale consists of links from question phrases to columns, value conditions attached to columns, and explicit "not present" claims for concepts it believes the database lacks. It then emits an answerable or unanswerable label.
The third part is a verifier, which is ordinary code. It checks that each linked column exists, that each value condition fits the column type, and that each cited value occurs in the column, using a lookup query. A "not present" claim is tested by retrieving the most similar column descriptions and flagging the claim if a strong match exists. The value-occurrence check partly addresses SQD's execution-dependent failures.
Failed claims are returned to the model once for revision. If failures remain, the label is overridden toward unanswerable, and the count of failed claims is exposed as a confidence score.
Training follows DynaGuard's recipe: supervised fine-tuning on rationales, then group relative policy optimization, a reinforcement-learning method that compares groups of sampled outputs for the same input. The reward combines label correctness with the fraction of verified claims. Rationales are therefore rewarded for being checkable and correct, not only for agreeing with the label. Reference rationales for the supervised stage are derived from the gold SQL in EHRSQL. The gold query tells us exactly which columns and values an answerable question uses, so claim-level gold labels come for free.
The language-specific factor is a controlled swap. At matched parameter scale where possible, the classifier backbone is either a multilingual model such as Llama-3.1-8B or Qwen, or a language-specific model: TigerLLM-9B for Bengali, NorGLM for Norwegian, an Arabic-centric model from the ArabicMMLU study, and SeaLLMs for Thai. Data, rationale format, verifier and training budget are all held fixed. Because the verifier scores linking claims individually, we can tell whether local pre-training helps specifically at the step of mapping local-language phrases onto English schema terms, not just in overall accuracy.
Baselines
The naive reference point is the SQD classifier with its schema tokens removed. On EHRSQL-2024 this gives F1 0.5739 (SQD). A direct-prompting row comes from the same paper: prompted GPT-4o at F1 0.7778 and LLaMA3.1 at 0.5487. These tell us whether translated questions push weak systems toward chance.
The strongest existing method is SQD itself, at F1 0.8547, AUC 0.9062 and accuracy 0.9426 on English EHRSQL-2024. It is the strongest competitor because it already exploits schema descriptions and is cheap. It may simply transfer if its encoder is swapped for a multilingual one. We will re-run it with a multilingual sentence encoder, training on the same English plus machine-translated training questions our method sees. We will use the same audited descriptions and tune its threshold on the same validation split. Beating it means higher F1 on the non-English test sets with bootstrap confidence intervals that exclude zero, while staying within one F1 point on English. No reported result was found on Lacuna for SQD or any other method on non-English EHRSQL, so our re-run is the reference.
The second strong competitor is translate-test. Each question is machine-translated to English, and English SQD is applied. It will be tuned identically.
The reasoning-before-labeling baseline is a DynaGuard-style model trained on the same data and budget, with free-form chain-of-thought and no verifier. DynaGuard reports 73.1% F1 on DynaBench, but no reported result was found on Lacuna for it on EHRSQL.
Three ablations isolate the key ingredient. The first uses the structured rationale format with no verification, which tests whether format alone helps. The second uses verification without the revision step. The third uses our full system with unaudited descriptions. If the free-form and unverified-structured variants match the full system, verification is not doing the work.
On MASSIVE-AMR, the reference is GPT-4's roughly 24% detection of disallowed relations with an oracle relation list, and a deception rate above 80% without one (MASSIVE-AMR).
Why not just…?
Why not just use SQD with a multilingual encoder? This is the most serious threat. SQD's relevance scoring does not produce text, so it cannot hallucinate text. If it matches our system across languages, the honest conclusion is that rationales buy explanation but not accuracy. That would still be worth reporting, with verified-claim rates serving only as audit logs. Our bet rests on SQD's documented failure on value-dependent questions, which a relevance score cannot see. We will report results separately on the subset of questions whose answerability depends on stored values, where the prediction should be clearest.
Why not just translate everything to English first? On translated EHRSQL, translating back may be nearly lossless, making translate-test very strong. This is exactly why we add native-written questions, described below. MR. Guard reports that direct translation misses dialectal and culturally specific phrasing, but that finding comes from safety data. If translate-test ties us on native-written questions, the multilingual component of the claim fails.
Why not just prompt for chain-of-thought, or ask the model to check itself? BenHalluEval found free chain-of-thought degraded results in 12 of 21 settings. MASSIVE-AMR found self-checking against an allowed-relation list catches only about 24% of hallucinations. The free chain-of-thought and unverified-structured ablations test this directly on our data.
Why not just use a bigger multilingual model? Evidence that scale is not sufficient exists but is indirect. Granite Guardian 3B beat its 5B sibling in GuardSet-X, and TigerLLM-9B was competitive in Bengali. We include a 70B-class multilingual model as the classifier, run with the same verifier, to settle this. If it dominates the local 7–9B models on linking accuracy, the local pre-training hypothesis fails for this task.
Proposed Experiments
The main benchmark is a multilingual EHRSQL-2024. We will machine-translate and then human post-edit 1,000 test questions each into Bengali, Norwegian, Arabic and Thai, preserving EHRSQL's high share of unanswerable questions. To guard against translationese, native speakers will also write about 300 questions per language from clinical intents without seeing the English text. Labels for these come from the annotators writing gold SQL or marking a question unanswerable.
We add two external checks. The first is MASSIVE-AMR, recast as classifying whether a question is answerable with the allowed DBpedia relations, across its 50+ languages grouped by resource level. The second is StatBot.Swiss, whose bilingual open-data schemas are not translated. No reported result was found on Lacuna for unanswerable-question detection on StatBot.Swiss.
We report F1, AUC and accuracy for each method. We also report citation precision: the share of rationale claims the verifier confirms, scored against claim labels derived from gold SQL. We report the description audit failure rate by language and by writer model. Finally, we report calibration of the failed-claim count, measured as AUC for predicting misclassification, with the confidence-score methods benchmarked in MUCH (CCP ROC-AUC 0.772 there) as a reference style. Latency per question is reported for every method, since reasoning guardrails are known to be slow.
The decisive checks are four. First, the full system must beat multilingual SQD and translate-test on native-written questions, with the margin growing from Norwegian toward Thai and Bengali. Second, the no-verification ablation must lose most of that margin. If it does not, verification is not the cause. Third, the failed-claim count must predict errors better than the model's own label probability. Fourth, with backbone size held fixed, local models must show higher citation precision on linking claims than multilingual ones; otherwise the language-specific pre-training advantage does not extend to schema linking. Any one of these failing falsifies a named part of the claim.
References
- BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali
- Leveraging LLM-Generated Schema Descriptions for Unanswerable Question Detection in Clinical Data
- DynaGuard: A Dynamic Guardian Model With User-Defined Policies
- MR. Guard: Multilingual Reasoning Guardrail using Curriculum Learning
- MASSIVE Multilingual Abstract Meaning Representation: A Dataset and Baselines for Hallucination Detection
- Multilingual Hallucination Gaps in Large Language Models
- NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian
- SeaLLMs -- Large Language Models for Southeast Asia
- ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
- Columbo: Expanding Abbreviated Column Names for Tabular Data Using Large Language Models
- ITUNLP at SemEval-2025 Task 8: Question-Answering over Tabular Data: A Zero-Shot Approach using LLM-Driven Code Generation
- AlphaPro at SemEval-2025 Task 8: A Code Generation Approach for Question-Answering over Tabular Data
- GuardSet-X: Massive Multi-Domain Safety Policy-Grounded Guardrail Dataset
- StatBot.Swiss: Bilingual Open Data Exploration in Natural Language
- MUCH: A Multilingual Claim Hallucination Benchmark