Measuring Functional Equivalence for Idioms by What Readers Can Answer

When a Chinese passage says the tomb trees "have arched" (墓木已拱), a good English translation has to do two things that pull against each other. It must get across the meaning, which is that a long time has passed since someone died. It may also need to keep the image of trees grown over a grave, depending on who the translation is for. Translation theory names both pressures. Functional equivalence asks the target text to produce the same effect on its readers as the source produced on its own readers. Skopos (purpose-driven) theory says the right strategy depends on the translation's job and audience. Hybrid strategies keep some foreign elements and naturalize others. Machine translation research has shown that systems fail on exactly this material, but it has no way to check the theory's central claim. Nobody measures whether target readers end up understanding what source readers understood. Systems are scored against reference strings, or by metrics that fail on idioms.

We propose to make functional equivalence measurable and then build against that measurement. For each figurative expression, we split what a reader should take away into two parts. The tenor is the meaning being conveyed. The vehicle is the image or cultural content carrying it. We write short comprehension questions (probes) for each part. A translation is functionally equivalent for a stated purpose if monolingual target readers answer the tenor probes about as well as source readers do, and answer the vehicle probes at the level that purpose requires. They should also not falsely attribute target-culture imagery to the source. With this yardstick in place, we test a simple question that matters in practice. Does an explicit per-idiom decision about what to do with the image beat simply telling a large language model (LLM) the purpose of the translation? The possible decisions are keep it literally, keep it with an inline gloss, split it, replace it with a target idiom, or drop it.

Core Claim

We make two claims. The first concerns measurement: reader-probe scores will rank systems differently from the automatic metrics now used for idiom translation, especially on the vehicle axis, which no current metric represents. The second concerns method: planning the vehicle treatment explicitly under a purpose brief and a length budget, then choosing among candidates by simulated-reader probe accuracy, will give translations whose image retention can be controlled at fixed meaning accuracy. Purpose-brief prompting alone will not achieve this. We treat the second claim as falsifiable. If brief-only prompting matches the planned system on both axes, the contribution becomes the measurement and the finding that LLMs already follow purpose briefs. We will report it that way.

Where This Sits

The motivating evidence comes from three places. First, idiom failure is persistent. A probing study of Marian translation models found that over 76% of figurative English idioms in MAGPIE (a corpus of idioms labeled as used figuratively or literally) were translated literally (Can Transformer be Too Compositional?). Evaluating LLMs on Chinese Idiom Translation built IDIOMEVAL, 900 annotated Chinese-English pairs, and found GPT-4 still erring on 28% of idioms. Second, strategy choice is uncontrolled. Evaluation of Machine-Translated Subtitles for the Documentary China found DeepSeek leaning foreign (89% literal retention of calendrical terms) and Moonshot leaning domestic (52% free translation). Both still failed on 墓木已拱. Third, evaluation is broken. IDIOMEVAL reports metric-human Pearson correlations below 0.48. Benchmarking MT Evaluation Metrics with Chinese MWEs finds metrics overrating multiword-expression translations, with discrepancies against human MQM scores across 31.2% of property groups. In Examining the Tip of the Iceberg, an <idm> tag raised idiom accuracy while lowering BLEU. All of these papers diagnose the problem. None defines what a correct non-literal translation must achieve for a given audience. That definition is what we add.

The translation-studies papers supply the candidate decisions rather than any algorithm. Integrated Adoption of Domestication and Foreignization shows, on a single poem, imagery kept literal while grammar is naturalized, at 1.65× length. On Chinese Translation of English Proverbs describes split-phrase renderings: literal first half, explanatory second half. It also argues against substituting target-culture figures (the "Speak of Cao Cao" case). Exploration of Translation Skills for Ceramic Culture under Skopos Theory describes transliteration plus in-text gloss. We turn these into the planner's action set, and the probes test whether each one actually works on readers.

Two papers constrain the design. CaMMT frames conserve-versus-substitute for culture-specific items and reports a 67.5% human preference for multimodal, culture-conserving outputs. This warns against any design that defaults to domestication. However, its items are short captions with depictable referents, unlike idioms. Do GPTs Produce Less Literal Translations? shows that LLMs already paraphrase idioms more freely: 18.08% unaligned source words, against 13.70% for a commercial translation system. The risk with strong LLMs is therefore silent loss of the image, not literalism. The vehicle probe is designed to catch that loss.

Research Program

The probe resource. We draw about 600 Chinese passages containing chengyu from CHENGYU-BENCH (Fu et al.), which has longer contexts than earlier sets, and from IDIOMEVAL's source sentences. We add the culture-loaded subset of the China documentary subtitles for a setting with a hard length limit. Bilingual annotators label each idiom's tenor and vehicle. They then write three multiple-choice probes:

  • a tenor probe asking what the passage implies;
  • a vehicle probe asking what image or cultural content the passage used;
  • a misattribution probe, with a distractor naming a plausible target-culture image (for example, "the devil").

Native Chinese readers answer the probes on the source to set the source-side baseline. For any translation, monolingual English readers who know no Chinese answer the same probes. We define tenor equivalence as $F_t = a_t^{\mathrm{tgt}} / a_t^{\mathrm{src}}$, where $a_t^{\mathrm{tgt}}$ and $a_t^{\mathrm{src}}$ are tenor-probe accuracies for target and source readers. Vehicle retention $V$ and misattribution rate $M$ are raw target-reader rates. We also run a scaled-down English-to-German version on MAGPIE idioms and the Fadaee German-English set, to check that the protocol works in another language pair.

Purpose briefs. We use three briefs, each a short natural-language description plus numeric targets.

Brief Priority Length budget Vehicle retention
Documentary subtitle High tenor accuracy Character cap Moderate
General young reader Tenor None Optional
Scholarly or literary High vehicle retention, low misattribution None High

These follow Nord's "function plus loyalty" reading of skopos theory (Introduction of Functionalism): purpose decides how much freedom the translator has, and loyalty to the source sets limits on it.

The method. The method has three steps:

  1. An LLM first marks each figurative span. For each span it chooses one of five actions given the brief and the remaining length budget: calque, calque with inline gloss, split-phrase, target-idiom substitution, or plain paraphrase.
  2. It then renders the sentence under that plan.
  3. We sample several plans and select the final candidate with simulated readers. A separate LLM, prompted to act as a target-language reader without Chinese and shown only the translation, answers the probes. The candidate that best meets the brief's targets within the length cap is chosen.

The simulated reader is the weak point. An LLM reader may decode a literal calque using cultural knowledge that real readers lack. We therefore calibrate it against human answers on a held-out slice. We report results with simulated-reader selection only if its agreement with human probe outcomes is acceptable. Otherwise we fall back to the trained IDIOMEVAL error detector for selection, and use the probes only for evaluation.

Baselines

The naive reference point is direct translation by the same LLM with a plain "translate this" instruction. This gives the system's built-in strategy bias. Based on the China subtitle study, we expect it to sit at one end of the vehicle axis.

The strongest existing system is GPT-4-class direct translation. On IDIOMEVAL the cited paper reports a 28% idiom error rate for GPT-4 (Yang et al.). Beating it means more than lowering that error rate. It means higher $F_t$ at equal or better $V$ and $M$, under every brief.

Purpose-brief prompting is the most dangerous competitor: the identical brief is given to the same model in the prompt, with no explicit plan and no reranking. No result for this setting has been reported on Lacuna; there is no reported result found for skopos-conditioned idiom translation.

The metric-reranking baseline uses best-of-N with the same N, choosing by COMET-QE or by the fine-tuned Qwen2.5 idiom-error detector. That detector reached 0.68 macro F1 in IDIOMEVAL. This isolates whether probe-based selection matters beyond choosing well by existing measures.

The ablations remove one component at a time: the plan step while keeping the reranker, the reranker while keeping the plan, and the brief while keeping a fixed action.

All systems use the same base model, the same passages, the same N, the same decoding settings, and the same prompt-tuning budget on a shared development split. Human probe evaluation is blind to system identity.

Why not just…?

Why not just put the purpose brief in the prompt? This is a real threat. LLMs paraphrase freely and may follow briefs well. The experiment settles it directly. For each brief, we plot $V$ against $F_t$ for brief-only and planned systems. If brief-only reaches the same point, explicit planning adds nothing and we will say so. We expect brief-only to under-retain images under the subtitle length cap, because nothing forces it to spend characters on the vehicle. This expectation is unproven.

Why not just rerank with COMET-QE or the idiom-error detector? These measures have no notion of the vehicle. They will happily select a fluent paraphrase that drops the image or swaps in a target-culture image. The 31.2% discrepancy rate on multiword-expression metrics and the correlation below 0.48 on IDIOMEVAL suggest they also misjudge the tenor. The metric-reranking baseline tests this directly.

Why not just tag idioms or use a dictionary of equivalents? The <idm> tag tells a model that an idiom is present, not what to do with it, and Fadaee et al. report that it cost overall BLEU. A dictionary gives one fixed rendering per idiom regardless of purpose. Dictionary substitution is in fact the target-idiom action in our planner. If it wins under some brief, the planner should learn to pick it there.

Proposed Experiments

The first decisive check is about measurement. Across all systems, we compute the rank correlation between human probe scores ($F_t$, $V$, $M$) and BLEU, COMET-QE, and the IDIOMEVAL detector. Claim one predicts low agreement on $V$ and $M$. If the metrics agree with the probes, the measurement adds nothing and we report that.

The second check is about control. We sweep the brief and the length cap on the China subtitle subset. We test whether vehicle retention moves monotonically with the brief while tenor equivalence stays above about 0.9 of the source-reader level. We compare against the fixed strategies that DeepSeek and Moonshot showed in that study.

The third check is the split-phrase and gloss actions under the subtitle cap. We measure how much of the 1.65× expansion seen in the hand-translated poem these actions avoid, and whether they keep $V$ without hurting $F_t$.

The last check is a robustness test. On the English-to-German MAGPIE subset, we test whether rankings carry over to a language pair where the base model is less literal.

A clean negative result would also be informative. Such a result would be one where brief-only prompting matches the planned system, but metrics and probes disagree. It would show that the bottleneck for figurative translation is evaluation, not generation.