Multimodal machine translation aims to use a picture to translate a sentence correctly. "Glasses" should become spectacles or drinking vessels depending on what the image shows. In practice, models trained on the standard data learn to ignore the picture. The training sentences already say everything the image shows, so on a contrastive test built around exactly this kind of ambiguity, text-only and most multimodal systems score close to coin-flip accuracy (Tackling Ambiguity with Images). Image–sentence–translation triplets are scarce, so the field generates synthetic ones. The filters used to clean that synthetic data keep pairs whose text and image strongly match, and those are precisely the redundant pairs that teach a model to ignore the image.
We propose to invert the filter. A synthetic triplet should be kept only if three conditions hold. The source sentence is short and ambiguous, the way real users write. A text-only translator cannot settle the ambiguity. Swapping the image for one depicting the other meaning flips which translation a multimodal model prefers. The bet is that two problems usually studied separately have a shared cause and are the same problem. The synthetic-to-real gap is that machine-written text is verbose where humans are terse. The visual-indifference problem is that images are redundant with captions. Verbose synthetic text disambiguates itself, so it removes the very cases where the image matters. A filter for "visual indispensability" therefore fixes both at once.
Core Claim
The claim is narrow and testable. Take a fixed base translator, a fixed budget of synthetic triplets, and a fixed training recipe. Selecting triplets by how much a contrastive image changes the translation should improve accuracy on real lexical-ambiguity benchmarks more than selecting by image–text similarity, by text-only ambiguity alone, or not filtering at all. It should cost little or nothing on ordinary descriptive translation.
Write $X$ for the source sentence, $Y$ and $Y'$ for its translations under the two senses, and $V$ and $V'$ for images depicting those senses. Let $p_{\text{mm}}$ be a multimodal translator's probability and $p_{\text{txt}}$ a text-only translator's. The indispensability score of a candidate is
$$ s(X,Y,V) = \big[\log p_{\text{mm}}(Y \mid X, V) - \log p_{\text{mm}}(Y \mid X, V')\big] \cdot \mathbb{1}\big[\,|\log p_{\text{txt}}(Y \mid X) - \log p_{\text{txt}}(Y' \mid X)| < \tau\,\big] $$
Here $\tau$ is a margin below which the text-only model counts as undecided. The first factor asks whether the image does real work. The second asks whether the text alone fails. The score is the triplet-level analogue of the contrastive evaluation in CoMMuTE, moved from test time into data curation.
Prior Work and How This Differs
Three systems motivate the gap.
Bridging the Gap between Synthetic and Authentic Images generates images from source text with Stable Diffusion. It aligns synthetic and authentic image features with optimal transport and a prediction-consistency loss. It applies this alignment to every generated image and never discards bad ones, and its text side is ordinary descriptive captions.
Make Imagination Clearer! fine-tunes the diffusion generator with reinforcement learning against scene-graph matching. This reduces wrong counts and relations, but it is expensive. It also optimizes the image to agree with the text, which is redundancy again.
LAMBDA filters patch captions by CLIP similarity and then has an LLM merge them into long composite sentences. Its pipeline therefore enacts both failure modes we target: similarity-based selection, and the verbosity documented by Towards Understanding Bias in Synthetic Data for Evaluation, which finds real queries much shorter than LLM-written ones.
Our proposal differs from all three in what the filter rewards. It keeps examples where image and text are individually insufficient, not examples where they agree.
Two papers supply our training recipe and data regime. Towards Zero-Shot Multimodal Machine Translation adapts NLLB using English image–caption pairs. It trains with visually conditioned masked language modeling and distillation toward the frozen text-only model, and uses classifier-free guidance at inference. We reuse this recipe unchanged and replace only the data. Beyond Triplet shows that partial triplets with a gated fusion still help. Its EMMT e-commerce set, where images are genuinely needed, gives us a real-world test domain.
Our verification design borrows from the consensus-filtering line, where agreement among heterogeneous models stands in for correctness: Ensemble-Instruct, and the Qwen-plus-InternVL vote filter in Chat-based Person Retrieval. That person-retrieval paper also shows that such cleaning raises downstream accuracy by about 2.6 Rank-1 points.
One finding constrains the design. Towards Understanding Bias in Synthetic Data for Evaluation shows that judges favor models from their own family. This is why the sense-to-image check uses models from different lineages, and why the multimodal scorer $p_{\text{mm}}$ must not be the same model we then train.
CoLLM supports the decision to make synthetic texts short. In composed image retrieval, it deliberately wrote multiple short modification texts to mimic human queries.
Research Program
The pipeline has four stages.
The first stage builds a sense inventory of polysemous English words that have distinct translations in French, German, and Czech. Every word appearing in CoMMuTE is excluded, so that the benchmark measures generalization rather than memorized words.
The second stage generates candidates. An LLM writes short, telegraphic source sentences of five to ten words that use the word without disambiguating context. It produces a sense-specific translation for each reading. A diffusion model renders one image per sense.
The third stage verifies the images. Two vision-language models from different families, plus CLIP as a third vote, must agree which image depicts which sense. Any candidate where they disagree is dropped. This removes generation hallucinations, the failure Make Imagination Clearer! fixes with reinforcement learning, at a fraction of the cost.
The fourth stage computes $s$ and keeps the top-scoring candidates. Our default scorer $p_{\text{mm}}$ is VGAMT from Tackling Ambiguity with Images; frozen NLLB serves as $p_{\text{txt}}$. We also apply the same score to real Conceptual Captions pairs, using retrieved contrastive images, to test whether indispensability filtering helps on non-synthetic data too.
The filtered set is mixed with the original Conceptual Captions data, and we train with the ZeroMMT recipe at a fixed number of examples and steps. We then examine three things: which sentence lengths and sense types survive the filter, how the score correlates with the text-only model's uncertainty, and whether the trained model's reliance on the image grows. Reliance is measured as the gap between its scores with correct and with swapped images.
Baselines
The naive reference is the ZeroMMT recipe on Conceptual Captions alone, with no synthetic data. Alongside it we run the same recipe with all generated triplets added unfiltered, at matched training steps. The unfiltered run answers whether the filter adds anything beyond the data itself. Bridging the Gap reports that naively adding synthetic images gave only marginal gains, so we expect this run to be weak, but it must be measured.
The strongest existing comparisons on the main metric are VGAMT, at 67.1% CoMMuTE accuracy as reported in Tackling Ambiguity with Images, and ZeroMMT, at 65.0 on English–French CoMMuTE with classifier-free guidance as reported in Towards Zero-Shot Multimodal Machine Translation. ZeroMMT is the fairest direct competitor, because it shares our base model and training recipe and differs only in data. We rerun it ourselves with an identical tuning budget for the guidance strength and learning rate, rather than relying only on the reported number. For standard translation quality, the reference is the 36.04 BLEU on Multi30K English–German Test2017 reported by Bridging the Gap. On EMMT it is the 46.55 BLEU of the fusion-plus-prompt system in Beyond Triplet.
Three selection-rule ablations each select the same number of triplets from the same verified pool and isolate the key ingredient. The CLIP-similarity rule keeps the pairs with the highest image–text similarity, which is the standard filter. The text-ambiguity-only rule keeps candidates where NLLB is undecided but never checks that the image flips the choice. The image-flip-only rule keeps the first factor of $s$ without the text-only gate. A further ablation keeps the full score but swaps our short sentences for LLM-default verbose ones, to test the claim that verbosity destroys indispensability.
Beating the field means three things. On CoMMuTE, our model should exceed rerun ZeroMMT and VGAMT's reported 67.1%, with the gap outside a paired bootstrap interval; with only 155 sentences per language pair, this interval will be wide, so we pool across languages. On EMMT, it should improve the ambiguous-word accuracy. On Multi30K, it should stay within about half a BLEU point of the no-synthetic baseline.
Why not just…?
Why not just filter by CLIP score, as everyone does? CLIP-similarity filtering selects pairs where the image restates the text. The visual-indifference result in Tackling Ambiguity with Images suggests this is exactly the data that produces near-chance contrastive accuracy. This is a hypothesis rather than an established fact, and the CLIP-similarity ablation tests it directly. If CLIP-filtered data matches ours on CoMMuTE, the core claim is wrong.
Why not just mine ambiguous sentences with the text-only model and skip the image check? This is the most serious threat, because it is cheap and may capture most of the signal. If the verified images are already correct, text-side ambiguity might be all that matters. The text-ambiguity-only ablation settles it. The image-flip factor earns its place only if it adds accuracy over this rule at equal data size. If the two rules perform within noise, we would simplify the method to the text-only rule and report that honestly.
Why not just scale text-only translation or use more unfiltered synthetic data? On CoMMuTE, text-only systems sit near 50% by construction, because the sentences carry no disambiguating context (Tackling Ambiguity with Images). Unfiltered synthetic images gave only marginal gains in Bridging the Gap. Our unfiltered-augmentation baseline, run at matched compute, checks whether quantity alone closes the gap in our setting.
Why not just fine-tune the diffusion model with scene-graph rewards instead of filtering? Make Imagination Clearer! improves image faithfulness this way, raising human-rated alignment from 1.44 to 2.68 out of 3. However, it rewards agreement between image and text, not the image's contribution to translation, and it requires reinforcement-learning updates to a diffusion model. Our cross-family agreement check handles faithfulness by discarding failures. Our score handles usefulness, which their method does not target.
Decisive Experiments
The main experiment trains the same NLLB-based model under every selection rule above. Each run uses the same synthetic budget, at 50k, 200k, and 1M kept triplets, and the same training steps. We evaluate on CoMMuTE in French, German, and Czech, plus the Arabic, Russian, and Chinese extensions from Towards Zero-Shot Multimodal Machine Translation. We also evaluate ambiguous-word accuracy and BLEU on EMMT, BLEU and COMET on Multi30K Test2016 and Test2017, and BLEU on Ambiguous COCO. We report Multi30K BLEU to compare against prior numbers, but we treat it as a regression check, since that benchmark rewards ignoring the image.
Three results would be decisive. The first is that indispensability-filtered data beats CLIP-filtered data on CoMMuTE at every budget. The second is that the full score beats the text-ambiguity-only rule, which tells us whether the image check matters. The third is that switching to verbose source sentences sharply reduces how many candidates pass the filter and how much they help. That would confirm the link between the synthetic-to-real text shift and visual redundancy.
A supporting diagnostic measures each trained model's image reliance on held-out ambiguous sentences, comparing correct images against swapped ones. If our gains are real, this reliance should rise. If accuracy improves while reliance stays flat, the gains come from text-side effects and not from the mechanism we propose.
References
- Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation
- Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation
- Make Imagination Clearer! Stable Diffusion-based Visual Imagination for Multimodal Machine Translation
- LAMBDA: Large Language Model-Based Data Augmentation for Multi-Modal Machine Translation
- Towards Understanding Bias in Synthetic Data for Evaluation
- Towards Zero-Shot Multimodal Machine Translation
- Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translation
- Ensemble-Instruct: Instruction Tuning Data Generation with a Heterogeneous Mixture of LMs
- Chat-based Person Retrieval via Dialogue-Refined Cross-Modal Alignment
- CoLLM: A Large Language Model for Composed Image Retrieval