Small neural machine translation (MT) models are cheap enough to run on a laptop or phone. Their weak spot is figurative language. Given "spill the beans," they tend to translate each word and produce a sentence about legumes. The expensive fix is to send everything to a large language model, which is costly and still imperfect. The obvious cheaper fix is to detect idioms and send only those sentences to the large model. This note argues that detection asks the wrong question. The useful question is whether this particular small model has just translated this particular expression literally. Prior mechanistic work suggests the small model's own attention shows whether it treated a phrase as a single unit or built it up word by word, and that signal is available at no extra cost during decoding.
We propose a two-tier translator built on that signal. A small model translates everything. A lightweight probe reads the small model's internal state and predicts whether it produced a literal mistranslation. Only flagged sentences go to a large model. We evaluate on text with the naturally low density of idioms found in real documents, not idiom-packed test sets, because that density determines both the compute savings and the cost of false alarms. We treat the first round of results as an explicit pilot with calibrated human judgments and a pre-stated go/no-go rule, because this subfield's standard automatic metrics are known to mislead.
Core Claim
A router that predicts failure should reach the same figurative-translation quality as a router that detects idioms, while escalating fewer sentences. It should also catch figurative expressions that no idiom lexicon lists. The claim rests on the analysis in Can Transformer be Too Compositional?, which examined Marian-MT translating English MAGPIE idioms into seven languages. It found literal output for over 76% of figurative idioms. It also found that the successful cases carry a distinctive signature. The encoder binds the idiom tokens tightly to one another and attends less to surrounding context. The decoder attends less to the source and more to the end-of-sentence token. Erasing idiom features causally reverted 27–40% of figurative translations back to literal ones.
Dankers et al. used this signature to explain behaviour. We use it as an online warning light that routes compute. A second piece of evidence points the same way: Predicting Human Translation Difficulty with Neural Machine Translation found that translation surprisal (how unexpected the output words are given the source) and attention withdrawal from context predict human translators' effort. Internal model states are therefore reasonable proxies for difficulty. That study used general text, not idioms, and did not use these states to allocate compute.
Research Program
We first build the failure probe. On the training portions of the English–French/Spanish idiom corpus from Automatic Evaluation and Analysis of Idioms in Neural Machine Translation, we decode with the small model and label each idiom-bearing output as literal or not. The labels come from the LitTER blocklist procedure in the same paper, which counts word-by-word renderings of idiom components that are absent from the reference. We then extract, per source sentence, the Dankers features for candidate spans. These are within-span encoder self-attention, span-to-context attention, and cross-attention mass on the end-of-sentence token. We add translation surprisal as in Lim et al. and fit a small logistic classifier.
We test two ways of proposing candidate spans. The hybrid version uses an idiom lexicon. The detector-free version scans for contiguous source n-grams whose internal attention forms an unusually tight block. On MAGPIE, the literal-use annotations let us check that the probe does not fire on idiom strings used literally.
The escalation tier is a larger open model prompted few-shot. We choose few-shot because Creative and Context-Aware Translation of East Asian Idioms with GPT-4 found it maximized faithfulness among 27 strategies. The historical resource-constrained systems in Towards Computer Systems for Conversing in Polish motivate the multi-pass design, a cheap first pass with selective rework, but supply no technique.
The ecological-validity direction enters as the evaluation regime. Semantic Representations during Language Comprehension Are Affected by Context showed that isolated-sentence paradigms misstate how comprehension works in continuous text. Idiom benchmarks make an analogous mistake for routing: in an idiom-dense set, every false positive is cheap and every sentence is worth checking. We therefore report every result twice. The first is on the idiom test splits. The second is on full generic document streams, namely the WMT14 English–French and WMT13 English–Spanish test sets used by Baziotis et al., where idioms occur at their natural rate.
As a secondary psycholinguistic check, we ask whether the probe score predicts human translators' production duration on multiword-expression segments in the CRITT translation-process database beyond translation surprisal alone. We use linear mixed models as in Lim et al. This connects to The Idiom Processing Advantage is Explained By Surprisal, which attributed humans' faster idiom reading to low surprisal. Our question is whether a model's failure to chunk an idiom tracks human effort better than surprisal does.
Baselines
The first naive reference point is to send nothing onward: the small Marian model alone, which fails on over 76% of MAGPIE figurative idioms according to Dankers et al. The second is to send everything: every sentence goes to the large model. This sets the quality ceiling and the compute maximum. For context on that ceiling, Evaluating LLMs on Chinese Idiom Translation reports that GPT-4 errs on 28% of Chinese idiom translations. All routers share the same two translation models, decoding settings and prompt. They differ only in which sentences they escalate.
The strongest competitor is presence-based routing: an idiom lexicon match followed by a literal-versus-figurative classifier trained on MAGPIE, which escalates any sentence judged to contain a figurative idiom. This is the automated replacement for the oracle marker token in Examining the Tip of the Iceberg. That paper reports that its oracle token raised word-level idiom accuracy from 67.8% to 73.2% on German–English, while BLEU fell from 24.8 to 22.5. For an automated presence router in cascade form we found no reported result on Lacuna, so we build it ourselves. It gets the same training data and the same tuning budget as our probe.
A general difficulty router is also strong. It uses sentinel-src-24 from Estimating Machine Translation Difficulty, which the file reports as the best difficulty ranker on WMT24 data; no reported numeric result was found on Lacuna. We include the text-length heuristic that the same paper found beat LLM-as-a-judge, and a surprisal-only router, which isolates whether the attention signature adds anything beyond the cue from Lim et al. and Socolof et al.
The key ablation compares four variants: the probe with lexicon spans, the probe with attention-discovered spans, surprisal features only, and attention features only. All routers are compared by tracing escalation rate against quality. Beating the presence router means matching its LitTER and human-rated idiom accuracy at a significantly lower escalation rate on natural-density streams, or achieving better idiom quality at an equal rate.
Why not just…?
Why not just send everything to the large model? It costs the most compute and still fails. The 28% GPT-4 error rate in Yang et al. and the lower-resource-language drops in Memorization or Reasoning? show that scale is not a solution. Our send-everything baseline tells us how much of that ceiling the cascade retains per unit of compute.
Why not just detect idioms with a lexicon and classifier? This is the real threat. Failure prediction can only save compute on idioms the small model already gets right, which is under a quarter of MAGPIE figurative cases for Marian. If that were the whole story, presence routing would be nearly as good. We expect two further sources of advantage: expressions the lexicon misses (discontinuous or unlisted figurative phrases), and the higher success rates of stronger small models. We will measure both directly, including a sweep over small-model sizes. If presence routing ties on natural streams, we report that and do not claim the method.
Why not just check the output with a quality-estimation metric and re-translate low scorers? Tang et al. found COMETKIWI blind to figurative appropriateness. Benchmarking the Performance of Machine Translation Evaluation Metrics with Chinese Multiword Expressions found that metrics overestimate literal multiword-expression translations, so this approach should miss exactly the errors we target. We run it as an additional router to confirm.
Proposed Experiments
The pilot uses a few hundred idiom-bearing and natural-stream sentences in English–French and German–English. Before any confirmatory study we fix a go/no-go rule: the probe must beat presence routing on the escalation-rate–quality curve under LitTER. Human judgments are needed because surface metrics correlate below 0.48 with human idiom ratings according to Yang et al. We collect ratings of idiom adequacy from several bilingual raters and adjust them with the many-facet Rasch model from Correcting Human Labels for Rater Effects in AI Evaluation. That model separates each rater's severity from true quality, and the paper showed such adjustment can change system rankings when agreement is modest.
The confirmatory phase adds English into seven languages on MAGPIE, human error annotation on Chinese–English using the IDIOMEVAL taxonomy, and the CRITT duration analysis. The decisive checks all use natural-density streams. The main one is whether the probe reduces escalations relative to presence routing at equal calibrated human idiom accuracy. The others are whether the detector-free variant recovers figurative errors that the lexicon misses, and whether the probe stays quiet on literal uses of idiom strings in MAGPIE.
References
- Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation
- Predicting Human Translation Difficulty with Neural Machine Translation
- Automatic Evaluation and Analysis of Idioms in Neural Machine Translation
- Creative and Context-Aware Translation of East Asian Idioms with GPT-4
- Towards Computer Systems for Conversing in Polish
- Semantic Representations during Language Comprehension Are Affected by Context
- The Idiom Processing Advantage is Explained By Surprisal
- Evaluating LLMs on Chinese Idiom Translation
- Examining the Tip of the Iceberg: A Data Set for Idiom Translation
- Estimating Machine Translation Difficulty
- Memorization or Reasoning? Exploring the Idiom Understanding of LLMs
- Benchmarking the Performance of Machine Translation Evaluation Metrics with Chinese Multiword Expressions
- Correcting Human Labels for Rater Effects in AI Evaluation: An Item Response Theory Approach