Morpheme-Delta Evidence for LLM Judges of Agglutinative Translation

Large language models are increasingly used as judges of machine translation: they read a source sentence and a system's output and decide whether the translation is acceptable. These judges are least reliable exactly where help is most needed, in languages that build words by chaining many small grammatical pieces onto a root. In Turkish, Kazakh or Korean, a single wrong suffix can flip negation, case or politeness. Giving the judge more help sounds like the fix, whether that help is a reference translation, a dictionary or a rubric, but recent evidence shows extra context can bias judges toward penalizing correct alternatives.

We propose a narrower kind of help. A morphological analyzer splits the output into root and suffixes. The judge then sees only a short "delta": the handful of grammatical features where the translation's suffixes disagree with what the source requires, with uninformative entries filtered out. The lexicon behind that analyzer must stay current as new names and terms appear, so we also borrow from database maintenance. Specifically, we update the lexicon when measured drift crosses a threshold rather than on a fixed calendar, and we prune entries that later evidence contradicts. The two-sentence version is this: show the judge only the morphological disagreements, not the whole answer key, and keep the tool that finds those disagreements fresh at the lowest curation cost. Both halves are testable, and both can fail.

Core Claim

Our central claim is that, for translation into agglutinative languages, sparse morpheme-level disagreement evidence improves an LLM judge's detection of critical morphological errors more than three alternatives: no evidence, the judge decomposing words by itself, or a full reference or dictionary. We further claim it does so without the false-penalty bias that full references induce.

A secondary claim concerns the lexicon. When the language stream drifts, updating the lexicon on a drift trigger keeps judge accuracy close to an always-fresh oracle at a fraction of the human verification cost of frequent periodic updates. It also beats never updating, which degrades as unanalyzable roots accumulate.

The secondary claim is deliberately falsifiable. If the frontier judge's own knowledge of new vocabulary makes lexicon freshness irrelevant, we will report that and drop the update machinery.

Why this is not already done

The closest work is An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation. It has GPT-4o extract human-referring phrases and their gender before labelling a sentence, which raised Italian Set-N accuracy from 62.33% to 89.54%. Its decomposition is done by the judge itself, uses no external analyzer, covers only gender inflection in Indo-European languages, and never checks whether the intermediate extraction is correct. We move the decomposition to a verifiable external tool and target suffix chains.

Informative Evidence-guided Prompt-based Fine-tuning for English-Korean Critical Error Detection motivates the evidence half. It showed that dictionary lookups and round-trip translations added to a fine-tuned encoder's prompt lift Korean critical error detection from 0.4307 to 0.6833 Matthews correlation. Its evidence is static and dense, and the model is an encoder classifier rather than a generative judge.

Curse of Knowledge constrains the design. It found that references and rubrics help judges on clean samples (+4.74% for DeepSeek-v3) but hurt them on perturbed ones (−1.66%). It also documented "solution fixation", where judges penalize correct alternative solutions, and a ceiling on how many criteria a judge can track at once. Both findings argue for giving the judge evidence that is small and points at disagreements, rather than a full target to match.

How Reliable is Multilingual LLM-as-a-Judge? supplies the motivating failure. It reports average cross-model agreement of about 0.3 Fleiss' kappa across 25 languages, worse in low-resource languages and not fixed by scale or by the multilingual Aya-Expanse model. It attributes difficulty to resource level and never isolates morphology, which we do.

Research Program

The pipeline has three parts.

Delta extractor. For each target word, a morphological analyzer returns a root and a sequence of feature-bearing suffixes. Examples of features are negation, case, person, possession, tense and politeness level. For Kazakh this is Apertium-kaz, the rule-based segmenter whose brittleness on unknown roots Neural Machine Translation for English–Kazakh with Morphological Segmentation and Synthetic Data documented. For Korean it is an off-the-shelf morphological analyzer. Separately, a cheap LLM pass extracts, from the English source alone, the features each aligned phrase requires: polarity, number, honorific register implied by the addressee, and so on. The delta is the set of positions where required and realized features conflict.

Uninformative filter. Following the uninformative filter in Development and Evaluation of an Automated Annotation Pipeline and cDNA Annotation System, we discard delta entries that carry no signal. Such entries include features the source does not determine, analyzer outputs with several equally likely parses, and roots the analyzer could not parse. In that paper, filtering raised automated agreement from 49% to over 85%. The judge receives the source, the candidate, and at most a few delta lines phrased as questions rather than verdicts, for example: "the source is negated; the verb X carries no negative suffix — is negation expressed elsewhere?" It never receives a reference translation. This is our hedge against solution fixation.

Living lexicon. The analyzer's coverage depends on a root lexicon, and news-domain text constantly introduces new names, loanwords and terms. Those words become unanalyzable, so deltas go silent exactly where errors may hide. We keep a lexicon $L_t$ at time $t$ and track the unanalyzable-root rate over a sliding window $W_t$ of recent candidate words:

$$r_t = \frac{1}{|W_t|}\sum_{w \in W_t} \mathbf{1}[\mathrm{root}(w) \notin L_t].$$

An update fires when $r_t$ exceeds a threshold $\tau$. During an update, new root candidates are proposed by agreement between the analyzer's guesser, the LLM, and back-translation consistency, and only disagreements are routed to a human. Entries whose later usage repeatedly contradicts their stored analysis are pruned. This is the subtractive maintenance that Graph-based Fingerprint Update Using Unlabelled WiFi Signals showed matters under heavy churn: its forgetting and link-prediction components were worth 46.2% in location error when 90% of reference points disappeared. That paper also warns of a "freshness ceiling", where incremental updates accumulate drift until a full re-survey is needed. We measure whether such a ceiling appears for lexicons by tracking analyzer precision against periodic gold audits.

Baselines

The naive reference point is the judge with no evidence at all. GPT-4o and Qwen-2.5-72B will be prompted with source and candidate and asked for a critical-error label with rationale. A judge that beats this but not the stronger rows below adds nothing.

The strongest existing method on our main benchmark is the evidence-augmented encoder of Informative Evidence-guided Prompt-based Fine-tuning for English-Korean Critical Error Detection, which reports 0.6833 Matthews correlation on KoCED. We compare on the same KoCED test split with the same metric, using their reported number and our reimplementation if feasible. Beating it means exceeding 0.6833 overall and, more importantly, on the politeness and sentiment classes, where morphology carries the error. On the Kazakh challenge set we build, no reported result was found on Lacuna for LLM judges, so the comparisons there are internal.

The strongest trained metric is xCOMET. It flags error spans and labels them minor, major or critical, and it reports 0.96 AUROC on hallucination detection for high-resource pairs. We run it off the shelf on Korean and Kazakh and threshold its critical-span output. We tune the threshold on the same development split our judges' prompts are tuned on, with equal development-set access. No reported result was found on Lacuna for xCOMET on Korean or Kazakh.

The judge-only decomposition baseline adapts the phrase-level chain-of-thought of An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation. The judge segments target words and lists their features itself before labelling, with no external analyzer. The full-evidence baseline gives the judge the reference translation plus dictionary entries for all content words, which is the dense injection that Curse of Knowledge warns about.

Two ablations isolate our ingredients. The first removes the uninformative filter and passes all delta lines. The second replaces the question-phrased deltas with verdict-phrased ones ("negation missing"). All judge conditions share the same model, decoding settings, and a fixed budget of prompt-tuning iterations on the development split.

For the lexicon study, there are four comparators: a frozen lexicon, fixed weekly updates, fixed monthly updates, and an oracle updated at every window. Each costs the same per-entry human verification, so the policies are compared on accuracy per verification.

Why not just…?

Why not just give the judge the reference translation? References are the standard way to ground MT evaluation, and on clean data they might win; Curse of Knowledge itself shows a clean-sample gain. The risk is specific to rich morphology: a correct translation can choose a different but valid case frame or honorific construction from the reference, and fixation would penalize it. This is a real threat to our claim. We settle it with a valid-alternative set: correct translations, verified by native speakers, that differ morphologically from the reference. We report both error recall and the false-penalty rate on that set. If the full-reference judge matches our recall without raising false penalties, the delta idea loses.

Why not just let a strong judge decompose words with chain of thought? The 62.33%→89.54% jump in An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation shows that self-decomposition is powerful. The open question is whether a judge built on subword tokens segments Kazakh suffix chains correctly. We measure the judge's own segmentation accuracy against analyzer gold on a sample, which that paper never did. If self-decomposition is accurate and matches the delta condition, the external analyzer is unnecessary and we will say so.

Why not just use xCOMET? It is cheap, needs no prompting, and outperformed GPT-4 on span detection for high-resource pairs. Its weakness here is that its fine-grained supervision comes from human error-span annotations that barely exist for Kazakh, and it cannot absorb new vocabulary without retraining. Its head-to-head numbers on KoCED and our Kazakh set decide this directly.

Why not just skip lexicon updates, or re-annotate in a yearly batch? A frontier LLM may already know new Kazakh names, which would make an analyzer that is blind to them harmless. The temporal experiment tests this. We report judge accuracy restricted to sentences containing post-cutoff roots under the frozen, periodic, triggered and oracle policies. If the frozen lexicon loses nothing there, the update component is cut.

Proposed Experiments

Data. The first benchmark is KoCED, focusing on its politeness and sentiment classes. The second is a Kazakh morphological challenge set built in the style of Test Suite Evaluation: Morphological Challenges and Pronoun Translation, which revealed polarity flips that aggregate metrics hide. We perturb correct outputs on the WMT 2019 English–Kazakh test data and the news portion of the TIL corpus from A Large-Scale Study of Machine Translation in Turkic Languages, introducing single-suffix errors in negation, case, possession and person. We pair these with the native-verified valid-alternative set, and hold out a smaller Azerbaijani set from TIL to test transfer to a related language. Real system outputs from English→Kazakh models are labeled by native speakers for ecological validity, since TIL reports that human preferences diverge from BLEU on Turkic output.

Temporal stream. We collect Kazakh news over several months, translate it with fixed MT systems, and replay it in time order through the lexicon policies.

Metrics. We report Matthews correlation and per-class recall for error detection, false-penalty rate on valid alternatives, and the cross-judge agreement of How Reliable is Multilingual LLM-as-a-Judge? in Fleiss' kappa across GPT-4o, Qwen-2.5-72B and Llama-3.3-70B. For the lexicon study we add accuracy on post-cutoff-root sentences and the number of human verifications.

Decisive checks. There are three. First, delta evidence must beat both judge self-decomposition and full references on morphological recall, while keeping false penalties on valid alternatives at or below the no-evidence judge. Second, delta evidence should raise cross-judge kappa, because sparse shared evidence ought to make different judges agree more than parametric guesses do. Third, the drift-triggered lexicon must match the oracle within a small margin on post-cutoff sentences with fewer verifications than weekly updates. Failing the first check kills the core claim, failing the third removes the maintenance component, and in either case the result itself, published with the challenge sets, is useful to anyone building judges for these languages.