The same English words often need different translations depending on what they are for. "Keep left" painted on a road is a command to drivers. In a hiking guidebook it is advice. In a product photo of a door hinge it may be an assembly step. A good human translator picks the target-language form, whether an imperative, a polite request, a terse sign convention, or a full sentence, by first working out the text's purpose and audience. This idea comes from Skopos theory, the functionalist school of translation studies. Machine translation systems almost never do this, and the text alone often gives no clue. Very often, though, a picture of where the text appears does give a clue: a street sign, a menu, a textbook page, two characters of different ages talking in a film scene. Work on visual disambiguation in machine translation has so far used images only to pick word senses ("crane" the bird versus the machine). We propose using images to infer the communicative purpose, and then translating to fit that purpose.
The method builds a short, structured "purpose card" from the image and source text. The card records what the text does (warn, instruct, describe, advertise, address someone), who reads it, the setting, and the social relation between speaker and addressee. A translation model is then conditioned on that card. The card is useful in two ways. It names the step where functional translation errors start. It also gives a reviewer who cannot read the source language something they can check: they can look at the image and the card and say "no, this is a warning sign, not a caption." That second property speaks to a long-standing weakness of human-in-the-loop translation, which is that monolingual post-editors have no way to verify meaning.
Core Claim
We make two claims and test them separately. The first concerns translation quality. On a contrastive benchmark where one source string has two correct translations that differ in function, conditioning on an image-inferred purpose card will beat both text-only translation and a vision-language model prompted to translate directly from the image. A vision-language model (VLM) is a large language model that also takes images as input.
The second claim concerns oversight, and it is the one we expect to hold even if the first is only partly confirmed. Monolingual target-language reviewers who can see and edit the purpose card will catch and fix more functional errors than reviewers who post-edit raw output, even when they also see the image.
The novelty is narrow and specific. Images are treated as evidence about the skopos (the purpose), not about word senses. The purpose sits in an explicit, editable intermediate representation. And the result is evaluated by whether the target reader would respond correctly, not only by surface overlap with a reference translation.
Where This Sits in Prior Work
The motivating evidence comes from functionalist translation studies. Peng's study of public signs (An Analysis on the Translation of Public Signs in Wuhan) shows that literal sign translations fail to trigger the intended behaviour. Wen's corpus analysis of aviation textbooks (Flexibility in the Translation of Aviation Technology English Texts) counts 81 added contrastive connectives and 70 added cause-effect markers that human translators inserted to suit learners. Wen also explicitly notes that no user outcomes were measured. Rothkegel's TEXAN system (Pragmatics in Machine Translation) formalized texts as "text acts" with an intent component, content, and a repertoire of linguistic forms. It argued that the same intent needs different forms in different languages. It was hand-built and never evaluated quantitatively. Our purpose card is a learned, image-informed descendant of TEXAN's intent component. We borrow the decomposition and not the rule system.
The method comes from visual disambiguation work. Three systems show that images can resolve lexical ambiguity once models are forced to use them:
- Tackling Ambiguity with Images adds adapters to a frozen mBART translation model. It reaches 67.1% on the CoMMuTE contrastive benchmark, against roughly 50% for text-only models.
- Towards Zero-Shot Multimodal Machine Translation reaches 65.0 on English–French CoMMuTE using only English image-caption data. It uses a penalty that keeps the model close to the original text-only system, plus guidance at inference time.
- Detect, Disambiguate, and Translate (DeDiT) is the closest prior work. It first detects ambiguous words, then asks a VLM for a visual rationale, then translates conditioned on that rationale. It reports a 25.16-point accuracy gain over earlier systems on CoMMuTE.
All three restrict "ambiguity" to which sense a word has. DeDiT's rationale is a free-text explanation about one word. Our card is a fixed-schema description of the whole text's function, and a reviewer can edit it.
Two papers constrain the design. Towards Visually-Guided Movie Subtitle Translation for Indic Languages found that always injecting video features hurt translation. Grounding only the worst 20–30% of segments, as chosen by an oracle that uses reference translations, gave 2–5% relative COMET gains. Structured attribute summaries (setting, gender, relation, honorifics) were more robust than free-form scene captions. That attribute summary is the nearest existing relative of our card. We reuse its structured design, apply it to functional categories beyond subtitles, and replace the oracle with a model-predicted decision about whether the purpose is underdetermined. Probing the Need for Visual Context shows that mismatched images push translation quality below the text-only baseline. Any card-based system therefore has to be robust when the image is uninformative.
On the human side, Enabling Monolingual Translators: Post-Editing vs. Options reports 35% (Arabic) and about 29% (Chinese) correctness for monolingual editors, against 61–66% for bilingual professionals. Editors failed mainly where they could not reconstruct meaning. We test whether an image plus an editable purpose card is a cheaper verification channel than Koehn's word-lattice options, which he found increase cognitive load.
Research Program
The first component is a benchmark, which we call SkoposPairs, modelled on the contrastive design of CoMMuTE. We collect about 300 short English source strings. Each string is paired with two naturally occurring images in which it plays different roles, such as a prohibition on a sign versus a phrase in a caption, a menu item versus an advertising slogan, an assembly instruction versus a product description, or one line of dialogue between peers versus the same line addressed to an elder. Each pairing gets two professional reference translations into French, German, Chinese and Japanese. Japanese and the Indic data supply politeness and honorific contrasts. Each pair also carries a gold purpose card.
Annotators are told to make the two references differ only where the function demands it, so that a system cannot score by chance. A third "neutral" image, unrelated to the text, tests robustness. With that image the correct behaviour is to fall back to the text-only reading.
The second component is the system. One VLM backbone (a 7B open model fine-tuned, and a large prompted model as a second setting) predicts the card from the image and source. The card's fields are text act, reader, setting, speaker–addressee relation, and a confidence value. A translation model is then conditioned on the card text. We train card prediction on gold cards from a training split. We also generate synthetic cards by asking the large model to describe function on Multi30K and the WMT'18 product-title images. Training includes cards built from deliberately mismatched images, so the translator learns to ignore low-confidence cards. This copies the idea behind the zero-shot paper's penalty for drifting from the text-only model: when the card is uncertain, the system should behave like the text-only system.
The third component is a human study. We recruit monolingual target-language reviewers in the style of Koehn's study. Each reviewer corrects outputs under one of three conditions: the output alone, the output plus the image, or the output plus the image and an editable card. In the third condition, editing a card field regenerates the translation. A plain fluency-oriented post-edit cannot fix a wrong mood or politeness level without knowing it is wrong; a card edit can.
Baselines
The naive reference point is a text-only translation model such as NLLB or Qwen-2.5-7B-Instruct, the text-only baseline in the Indic subtitle paper. By construction it gives the same output for both images of a pair, so it should score about 50% contrastive accuracy on SkoposPairs. That matches the roughly 50% that text-only models score on CoMMuTE in the VGAMT and ZeroMMT papers. Any system below this level is harming translation.
The strongest competitor is end-to-end VLM translation: the same VLM backbone, given the image and asked to translate the text appropriately for its context. This is the most direct threat, because large VLMs may already infer function implicitly. A second strong competitor is a DeDiT-style pipeline, run as prompting with the same backbone, using DeDiT's detect–rationale–translate prompts unchanged. On CoMMuTE, DeDiT prompting reports a 25.16-point gain in disambiguation accuracy over prior systems. VGAMT reports 67.1% and ZeroMMT reports 65.0% on English–French; no absolute DeDiT accuracy figure was found on Lacuna.
All systems share the backbone, the training data (including the synthetic cards, which are offered to the fine-tuned end-to-end baseline as extra training text), and the tuning budget, measured as equal prompt-search trials and equal fine-tuning steps. To count as beating these baselines, our system must reach statistically higher contrastive accuracy on SkoposPairs, judged by a paired bootstrap over items. It must also not regress on CoMMuTE, Multi30K COMET or the neutral-image split. If it does not beat them, the first claim fails, and we will report that.
Three ablations isolate the card. The first replaces the structured card with free-text rationale of equal length; this tests whether structure matters, mirroring the attribute-summary versus scene-caption comparison in the Indic paper. The second predicts the card from text alone with no image; this tests whether the image carries the purpose signal. The third gives the translator the gold card; this is the upper bound and shows how much error comes from purpose inference versus purpose-conditioned generation.
Why not just…?
Why not just prompt a strong VLM to translate with the image in view? This is a real threat, and we treat it as the main hypothesis to beat rather than something to argue around. Nothing in the file shows end-to-end VLMs failing at functional adaptation. DeDiT's own results suggest that structured, staged reasoning helps on lexical ambiguity, but that is indirect evidence. If end-to-end prompting matches the card system on SkoposPairs, the first claim is false. The proposal still stands on the second claim, because an implicit inference cannot be inspected or corrected by a reviewer who cannot read the source. The human study measures exactly that.
Why not just tell the model the purpose in text? Where a client states the purpose, this is correct, and the gold-card ablation is exactly this condition. But signage photos, product listings and film scenes rarely come with a stated purpose. The gap between the gold card and the card predicted from text alone measures how often text fails to reveal the purpose. The gap between cards predicted with and without the image measures how much the image recovers.
Why not just caption the image and append the caption to the source? Elliott et al. showed in Multi30K that independent image descriptions dwell on salient scene details unrelated to the text. The Indic subtitle paper found free-form scene captions less robust than structured attributes. We nonetheless run a caption cascade with the same backbone, because the free-text-rationale ablation alone does not rule it out.
Why not just extend DeDiT's ambiguity detector to more words? DeDiT asks which word is ambiguous. Many functional failures involve no ambiguous word at all: the mood of a verb, the politeness level, whether to add a connective, the convention for a sign. The SkoposPairs items are built so that every content word has a single clear sense. If DeDiT nonetheless performs near our system there, then functional choice reduces to word-level choice more than we expect, and we would say so.
Proposed Experiments and Decisive Checks
The primary experiment reports contrastive accuracy on SkoposPairs across four language pairs for all baselines and ablations, with human functional-fitness judgments on a held-out subset. In those judgments, bilingual raters read each translation alongside its image and decide whether a target reader would respond as intended, for example by stopping, assembling a part correctly, or perceiving the right politeness level. For the sign and instruction items, a small behavioural check asks target-language readers, who do not see the source, what action the sign asks them to take. This answers Wen's call for outcome measures rather than retrospective description.
Secondary checks guard against regressions. We report CoMMuTE contrastive accuracy, compared against VGAMT's 67.1% and ZeroMMT's 65.0. We report Multi30K COMET; DeDiT's fine-tuned LLaVA reports 93.06. On the Indic subtitle set, we test whether a predicted "purpose underdetermined" signal can replace the oracle selection and recover some of the 2–5% relative COMET gain. On the WMT'18 product-title data, we test whether card disagreement predicts MQM errors, compared against the multimodal QE result of Pearson 0.602 from Multimodal Quality Estimation.
The human study reports, for each condition, the share of functionally correct final translations and the time spent per item, with Koehn's 29–35% monolingual correctness as historical context rather than a directly comparable figure.
Three results would settle the proposal:
- Predicted cards beat end-to-end prompting on SkoposPairs, and the image-free card falls well short. This would show the image carries purpose information and that making purpose explicit helps.
- The gold card far exceeds the predicted card. This would show purpose inference, not generation, is the bottleneck.
- Card-editing reviewers fix substantially more functional errors than image-only reviewers in equal time. This would validate the oversight claim regardless of how the first comparison turns out.
References
- An Analysis on the Translation of Public Signs in Wuhan
- Flexibility in the Translation of Aviation Technology English Texts
- Pragmatics in Machine Translation
- Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation
- Towards Zero-Shot Multimodal Machine Translation
- Detect, Disambiguate, and Translate: On-Demand Visual Reasoning for Multimodal Machine Translation with Large Vision-Language Models
- Towards Visually-Guided Movie Subtitle Translation for Indic Languages
- Probing the Need for Visual Context in Multimodal Machine Translation
- Enabling Monolingual Translators: Post-Editing vs. Options
- Multi30K: Multilingual English-German Image Descriptions
- Multimodal Quality Estimation for Machine Translation