Catching Suffix-Level Hallucinations with Finite-State Claim Units

When a language model answers a question from retrieved documents in Finnish or Turkish, much of what makes a statement true or false sits inside single words. Negation, tense, hearsay ("it is reported that…"), conditionality and grammatical role are marked by suffixes, not by separate words. The model can copy the right names and roots from the evidence and still flip a suffix. It can say someone was born in Turku when the source says he is said to have been born there, or turn a negated verb into an affirmed one. Current hallucination checkers have two weaknesses here. They first cut the answer into "atomic facts" with a model that tends to drop these qualifiers, and they then judge each fact with an entailment classifier that sees the right roots and accepts it. We propose to build the verifiable units from a finite-state morphological analyzer instead, and to compare the meaning-bearing features of each unit against the evidence directly. A finite-state analyzer is a hand-written grammar compiled into an automaton that maps any surface word to its lemma plus exact feature tags.

Our claim, in two sentences, is this. In morphologically rich languages, a large and checkable class of grounded-generation errors consists of operator errors: wrong polarity, tense, evidentiality, mood or case role on a correctly copied root. An analyzer-defined, scope-preserving claim unit, verified feature by feature, should catch these errors far more reliably than language-model decomposition plus an entailment classifier. The study is designed so that it can also show that this class is too small to matter, which would itself be useful to know.

Core Claim

The research file points to the gap from three directions. Multilingual Hallucination Gaps in Large Language Models extends FACTSCORE-style atomic-fact checking to 19 languages and reports factual precision falling from about 73.7% for very-high-resource languages to 56.9% for low-resource ones. It does not ask which kinds of errors drive the drop, and it relies on a Mistral-7B judge whose cross-lingual reliability is unexamined. Resolving Conflicting Evidence in Automated Fact-Checking shows in English that sentence-level evidence units lose the qualifying conditions a verifier needs, while paragraph units add noise. Granularity is therefore a real factor, but that work only compares sentences against paragraphs. Ask, Assess, and Refine shows that models correct themselves only when an external metric tells them what is wrong: unguided refinement dropped QAMPARI correctness recall from 18.6 to 4.8. Its feedback, however, comes from an English entailment model.

Our proposal sits between these. The unit of verification should be neither a sentence nor a model-written atomic fact. It should be a clause-sized unit whose boundaries come from morphology, which in these languages marks most clause boundaries through participles, converbs and nominalizations. Each unit should explicitly inherit the operators of the clauses that govern it, which is the "context-preserving" part. Verification then becomes partly symbolic. Roots are aligned through lemmas, and operator features are compared exactly, with the analyzer's ambiguity handled explicitly rather than guessed away.

This deliberately does not repeat two nearby ideas. Tokenization as Finite-State Transduction composes grammars with the tokenizer to constrain decoding to valid, canonical token sequences. That technique cannot catch our target errors, because a suffix-flipped hallucination is a perfectly well-formed word, and a morphological grammar would happily allow it. Evaluating Morphological Compositional Generalization in Large Language Models found that feeding morpheme-aligned tokens did not repair models' morphological failures. We take that as a constraint on design: the failure is not in the input segmentation, so we leave the model's tokenizer alone and put the morphological knowledge outside the model, in the checker.

Research Program

The first component builds claim units. Both the generated answer and the retrieved passages go through a two-level analyzer in the tradition of Koskenniemi and the Turkish description of Oflazer. We use Finnish and Turkish analyzers built with Foma (Hulden) or HFST. Every verb form, finite or non-finite, heads a unit. A small set of rules attaches each non-finite verb to the finite verb it depends on and copies down that verb's operators: matrix negation, reportative or "said/believed" frames, conditional mood, and evidential suffixes such as Turkish -mIş. Each unit is therefore a lemma-and-feature record of the form (predicate lemma, argument lemmas with case, inherited operator set). It still has a surface span, so a neural verifier can read it as text. This is the step that preserves context: a unit is short, but it never loses the qualifier that made the original sentence true.

The second component compares features. For each answer unit, we retrieve the evidence units whose predicate and argument lemmas overlap. Operator features are then compared under an ambiguity-aware rule. A word that receives several analyses (the Chukchi analyzer of Andriyanets et al. reports 1.43 per token) is flagged as a mismatch only if no pair of analyses agrees on the operator. Pairs where some analyses agree and others do not are labelled "uncertain" rather than forced either way. When roots are absent from the lexicon, which was the cause of 75.2% of the Chukchi analyzer's failures, the unit is handed to the neural verifier and logged. This keeps the checker's precision measurable separately from its coverage.

The third component closes the loop. Following the metric-to-instruction recipe of Ask, Assess, and Refine, each confirmed mismatch becomes a precise corrective instruction, for example "the evidence presents this as reported, not witnessed; the answer states it as fact." The generator rewrites the answer, with at most two iterations as in that paper. Because the analyzer is bidirectional, the same transducer also generates the corrected surface form when needed. It also produces controlled test data: we can take a correct answer, flip exactly one operator, and regenerate a well-formed word, which yields minimal-pair hallucinations.

Baselines

The naive reference point is the generator with retrieval and no checking at all. Its errors define the hallucination rate that every other system must reduce, and its natural outputs supply the error-type census described below.

The strongest existing competitor is language-model atomic-fact decomposition followed by a model-based or entailment-based judge, run in the target language. This is exactly the pipeline of Multilingual Hallucination Gaps in Large Language Models, and it is what a practitioner would use today. That paper reports aggregate FACTSCORE values of 73.7% (very high resource) and 56.9% (low resource) for generation quality. For hallucination detection accuracy on Finnish or Turkish there is no reported result on Lacuna, so we will measure this baseline ourselves. For fairness, it receives the same retrieved passages, the same decomposer and judge model family as our neural fallback, and an equal prompt-tuning budget on the same development split. We also give it a strengthened variant whose decomposition prompt explicitly asks it to keep negation, tense and hearsay qualifiers. Beating it means higher claim-level detection F1 on operator errors, measured on held-out natural errors and not only on synthetic flips, with no loss on entity errors.

A second strong competitor is the translate-to-English pipeline. Here the answer and evidence are machine-translated and checked with the English metrics of the ALCE suite (Enabling Large Language Models to Generate Text with Citations). It gets the same translation system for both sides and the same entailment model. On English ELI5, Ask, Assess, and Refine reports citation precision rising from 29.5 to 32.3 and MAUVE from 42.5 to 67.7 with its loop. We do not expect those numbers to transfer, and we report our own.

Three ablations isolate the ingredients. The first keeps the analyzer-defined units but drops operator inheritance, which tests whether context preservation matters or whether finer units alone do the work. The second keeps inherited units but verifies them with the entailment model only, which tests whether the exact feature comparison matters. The third replaces the finite-state analyzer with a neural morphological tagger trained on treebank data, which tests whether the finite-state machinery specifically matters.

Why not just…?

Why not just prompt the decomposer to keep qualifiers? This is the cheapest fix and a genuine threat, which is why its strengthened variant is a baseline. Two pieces of evidence make us doubt it will suffice. First, Evaluating Morphological Compositional Generalization in Large Language Models reports that GPT-4 reaches only about 51% coherence on Turkish words built from unfamiliar roots, against about 98% for native speakers. Reading stacked suffixes is exactly where these models are weak. Second, Ask, Assess, and Refine shows that models do not reliably audit their own outputs. If the prompted decomposer matches our detector on natural operator errors, the symbolic checker is unnecessary for these languages, and we will say so.

Why not just translate everything to English and use mature English checkers? English marks evidentiality lexically if at all, so a translation can silently turn "reportedly born" into "born" on both sides, or on one side only. Translation can therefore erase or create exactly the errors we target. The minimal-pair test settles this directly: we measure what fraction of single-operator flips survive round-trip translation as detectable differences.

Why not just use a neural morphological tagger instead of a finite-state analyzer? For Finnish and Turkish, where treebanks exist, a tagger may do as well. That is why it is an ablation, not a strawman. The finite-state case rests on two properties. The analyzer enumerates all readings, which our ambiguity-aware rule depends on, while a tagger commits to one. The analyzer also exists for languages with almost no annotated data, such as the Mexican languages served by Py-Elotl. If the tagger ties on Finnish and Turkish, our claim narrows to "operator-level checking matters," with finite-state analysis as the route for languages without treebanks.

Why not just pass whole paragraphs to the judge, since paragraphs beat sentences? Resolving Conflicting Evidence in Automated Fact-Checking found paragraphs better than sentences, but also found that more context added noise, and its study was in English. Paragraph-level judging is included as a baseline configuration. If it matches inherited units on operator errors, the case for fine-grained units collapses.

Proposed Experiments

Finnish is the primary language because it appears in both TyDi QA and Mr. TyDi, and those papers document morphology-driven mismatch such as consonant gradation. TyDi QA reports an mBERT minimal-answer F1 of 50.5 against a human estimate of 70.1. Mr. TyDi shows hybrid BM25-plus-dense retrieval improving MRR@100 by 8 points over BM25 on average. We use hybrid retrieval as the fixed retriever, so that retrieval quality is held constant. Turkish is the second language. Because no Turkish set appears in TyDi, we build a small grounded question-answering set from Turkish Wikipedia passages, using Turkish questions written by native speakers.

The first study is a census, and it is the main kill criterion. Native speakers annotate about 1,000 grounded answers per language from two open generators. Each unsupported claim is labelled as an entity error, a relation error, or an operator error (polarity, tense, evidential or mood, case role). If operator errors make up under roughly 10% of unsupported claims, the proposal's practical value is small, and we report that result.

The second study uses minimal pairs. The transducer generates single-operator flips of correct answers, balanced across operator types. We report detection precision and recall per type for our system, all baselines and all ablations.

The third study evaluates detection on natural errors. On the annotated census set, we report claim-level F1 broken down by error type, together with the rate of "uncertain" verdicts and the share of answer units that had to fall back to the neural verifier because of missing lexicon entries.

The fourth study measures correction. We run the feedback loop with finite-state-derived instructions against the same loop with entailment-derived instructions, using two iterations and the same generator. Outcomes are the unsupported-claim rate judged by humans and TyDi QA answer correctness. The loop must not trade operator fixes for new entity errors.

The decisive pattern we expect has three parts. The strengthened decomposition baseline should catch entity errors about as well as our system but miss a large share of natural operator errors. Removing inheritance should cost most of the gain on hearsay and negation-scope errors specifically. Feature-level feedback should lower the human-judged hallucination rate where entailment feedback does not. Any of these failing tells us which part of the idea was wrong.