Seeing Enough to Commit: Image-Licensed Grammatical Gender in Zero-Shot Multimodal Translation

English can say "the doctor" without saying anything about gender. German, Italian, or Arabic cannot: every determiner, adjective, and verb in the sentence must agree with a masculine or feminine noun. Translation systems resolve this forced choice by defaulting to the masculine, and they often do so even when the sentence says the doctor is pregnant. An accompanying photo can sometimes supply the missing information. A photo can also be uninformative: the person is turned away, it shows a group, or it shows no one. Appearance never establishes a person's gender identity. We propose a translation system that uses the image to decide whether it has enough evidence to commit to masculine or feminine morphology. When it does not, it produces gender-fair forms (neutral rewordings such as Lehrkräfte, or inclusive forms such as Bürger*innen) instead of the masculine. The system is trained only on English image–caption pairs and needs no multilingual image–text–translation data, which are expensive to collect.

The opportunity comes from two existing results. Speech translation already shows that a non-text signal can rescue models from the masculine default: end-to-end models use vocal cues to inflect correctly where text-only cascades fail (Gender in Danger?). Separately, ZeroMMT shows that a frozen text-only translator can be made to use images through adapters trained on English captions alone (Towards Zero-Shot Multimodal Machine Translation). No one has asked whether such English-only visual grounding transfers to grammatical gender. No one has asked whether grounding of this kind actually reduces masculine defaults or makes them worse, given that English-anchored multilingual vision–language models are known to amplify gender skew (Breaking Language Barriers or Reinforcing Bias?). And no one has replaced the binary masculine/feminine choice with a three-way choice of masculine, feminine, or uncommitted.

Core Claim

Masculine default in multimodal translation should be treated as a decision under uncertainty, not as a binary classification error. The model should commit to a gendered form only when the text or the image makes that commitment well supported. Otherwise it should fall back to gender-fair language, not to the masculine. The specific claim has two parts. First, a calibrated "commitment" signal learned only from English image–caption data can tell images where gender presentation is visually clear from images where it is not. Second, this signal can steer a frozen multilingual translator across target languages. On images where presentation is unclear, the combination should sharply reduce the masculine-default rate. On images where presentation is clear, it should match a strong visually-informed pipeline. It should also beat text-only gender debiasing, which cannot know when to abstain.

This differs from the closest work in three ways. CaMMT (CaMMT) motivates this proposal: it observes anecdotally that images help large vision–language models choose Arabic masculine or feminine morphology for nouns like "athlete". However, it treats gender as an incidental binary cue and never measures default rates, stereotype overrides, or abstention. ZeroMMT supplies our training recipe but evaluates only word-sense ambiguity, such as crane. Gender-Aware Contrastive Learning (Target-Agnostic Gender-Aware Contrastive Learning) supplies the steering mechanism. It is text-only and binary, and it has no notion of when evidence is insufficient.

Research Program

The base system is ZeroMMT. A frozen NLLB translator receives SigLIP image features through lightweight adapters. These adapters are trained on Conceptual Captions with two losses. The first is visually conditioned masked language modeling, which means predicting masked caption words from the image and the rest of the caption. The second is a KL penalty that keeps outputs close to the text-only model. We add three components.

The first component is an English-only commitment head. Many English captions describe people with gendered words ("a woman holding…") and others with neutral ones ("a person holding…"). Caption writers' word choices are a noisy proxy for how clearly gender presentation reads in the image. During masked language modeling we specifically mask person words. A small head on the adapter output then predicts a three-way distribution over masculine, feminine, and neutral word classes. We calibrate this head with temperature scaling on a held-out set. On that set, multiple annotators label perceived presentation with an explicit "unclear" option, and the targets are agreement rates rather than majority votes. The head never sees non-English text.

The second component is target-agnostic gender steering. Following GACL's finding that contrastively learned gender information in the shared encoder transfers across target languages, we learn a gender direction on the NLLB encoder states of occupational nouns. We learn it from German text only and apply it unchanged to French, Italian, Spanish, Russian, and Arabic. At inference time, if the commitment head's top class is masculine or feminine and its confidence exceeds a threshold $\tau$, we push the noun's encoder state along that direction. ZeroMMT's classifier-free guidance sets how much weight the image receives, so the visual pathway and the gender push can be tuned separately.

The third component is a gender-fair fallback. If confidence is below $\tau$ and the text carries no explicit gender trigger, the system does not steer toward masculine. For German, decoding is constrained toward the neutral or inclusive entries of the GeFMT dictionary, which lists 115 core nouns in masculine, feminine, neutral, and inclusive variants (GeFMT). For the other languages, standardized non-binary agreement does not exist, a limitation the binary-constraint literature stresses (Under the Morphosyntactic Lens). There the fallback is a doublet (masculine and feminine forms joined), and we report the doublet rate rather than claim neutrality. German is therefore the full three-way test bed, and the other languages test the binary and abstention behavior. Explicit text cues always take precedence over the image. This rule targets the failure documented in Investigating Failures of Automatic Translation in the Case of Unambiguous Gender, where models produce "Il mio dottore è incinto" ("my doctor is pregnant", with doctor and pregnant both in masculine form) despite a clear feminine trigger.

No existing benchmark pairs gender-ambiguous translation sentences with controlled images, so we build one, called WinoMT-V. We take WinoMT sentences (Evaluating Gender Bias in Machine Translation) and pair each one with real photographs retrieved from Flickr30k/COCO-style scenes depicting the occupation. Each photo is annotated by several people for perceived presentation, with an explicit "unclear" option. We prefer retrieval to image generation because generators would import their own stereotypes. Every sentence appears under four image conditions: clearly feminine presentation, clearly masculine presentation, unclear presentation or no person, and an incongruent image of an unrelated scene. The last condition is the laziness probe used by 3AM (3AM) and CaMMT. Throughout the work, perceived presentation is used only to license a grammatical form, never as a label of identity.

Baselines

The naive reference points are the frozen text-only NLLB model and an "always gender-fair" German policy that emits the GeFMT neutral or inclusive form for every occupation. The first policy shows the raw masculine default. The second scores perfectly on fallback rate by construction, but it should lose accuracy on clear-presentation images and on explicit-trigger sentences. Our system must beat both on a combined score.

The strongest text-only competitor is GACL applied to NLLB-200, which reports WinoMT accuracy rising from 61.7% to 78.2%, with spBLEU on FLORES-200 falling only from 40.5 to 40.0 (Target-Agnostic Gender-Aware Contrastive Learning). We retrain it on the same NLLB checkpoint and the same German contrastive data used for our steering direction. That keeps the gender supervision identical and leaves the image pathway as the only difference.

The strongest multimodal baseline is unmodified ZeroMMT, which reports 65.0 disambiguation accuracy on CoMMuTE English–French against a 50.0 text-only baseline. We run it with the same adapters, the same Conceptual Captions data, and a guidance-strength sweep of the same size as ours. No reported result was found on Lacuna for any multimodal system on WinoMT-style gender. For the ambiguous-context ceiling, text-only systems reach only 50% feminine accuracy on MuST-SHE's ambiguous category, versus 84.00% masculine (Test Suites Task).

We also run two further pipelines with equal tuning budgets for prompts and thresholds. One is a pipeline that classifies the image with a zero-shot SigLIP "photo of a man / woman" probe and then inserts a gender tag into the source text. The other is Qwen2.5-VL, prompted with the image and an instruction to use gender-fair forms when gender is unclear, which is the model family CaMMT evaluates. A text-only GPT-4-style five-shot prompt serves as the gender-fair reference: it reports a 77.78 Inclusivity Index on INES.

The key ablations remove one ingredient at a time. The first replaces the learned, calibrated commitment head with an uncalibrated binary probe and no abstention. The second removes the gender-fair fallback while keeping steering. The third trains the commitment head on all caption words rather than on masked person words.

Beating the baselines means three things. On unclear-presentation images, our masculine-default rate must be significantly lower than ZeroMMT's and GACL's. On clear-presentation images, accuracy must be within a few points of the tag pipeline. And chrF/COMET on Multi30K and FLORES-200 must show no larger drop than GACL's.

Why not just…?

Why not just classify the image's gender and tag the source sentence? This alternative is the real threat. On clear-presentation images it may match or beat us, and we expect that. It has no principled abstention, because an uncalibrated probe always outputs a confident binary label. It also inherits the English-anchor skew that every multilingual CLIP variant showed in Breaking Language Barriers or Reinforcing Bias?. We will add a thresholded version of the tag pipeline. If thresholded tagging matches our system on unclear images as well, the contribution narrows to the three-way decision rule and the benchmark, and we will report it that way rather than claim value for the learned head.

Why not just prompt a large vision–language model? CaMMT shows that Gemini and Qwen-VL already use images, and prompting works for inclusivity in text-only settings. However, prompt constraints leave measurable bias. AmbGIMT finds that bias "remains evident despite the presence of the constraints", with COMET dropping from 82.46 to 65.01 for non-binary subjects (Beyond Binary Gender). In A Tale of Pronouns, instruction following emerged only above roughly 11B parameters. We compare directly on WinoMT-V. If Qwen2.5-VL wins outright, the remaining argument for our approach is cost and frozen-translator coverage, and we will say so.

Why not just translate English captions and train supervised multimodal translation? The translation engine's masculine default would be written straight into the training targets (Evaluating Gender Bias in Machine Translation). We train this translate-train variant on NLLB-translated Conceptual Captions and expect a higher masculine rate on unclear images than ZeroMMT has.

Proposed Experiments

The decisive check is the four-condition WinoMT-V comparison. Our system succeeds only if three results hold together. First, its masculine-default rate on unclear images falls well below that of both ZeroMMT and GACL. Second, its accuracy on clear images rises above that of GACL, the text-only model. Third, swapping in incongruent images degrades it, showing it actually uses the image rather than producing a Multi30K-style near-zero sensitivity of 0.05.

A second check asks whether English-only grounding amplifies stereotypes. We measure the stereotype gap $\Delta_S$, the accuracy difference between pro-stereotypical and anti-stereotypical roles, under the unclear condition for ZeroMMT and for our system. A gap larger than the text-only gap would confirm the English-anchor concern for generative translation.

A third check uses Integrated Gradients, following A Tale of Pronouns, to compare the attribution placed on visual tokens with the attribution placed on explicit pronoun tokens. This tests whether the image ever gains enough influence to override occupational priors, and whether the text-precedence rule fixes the pregnant-doctor failure.

For German we also report how often the fallback produces gender-fair forms, using GeFMT categories with human fluency judgments. General translation quality is checked on Multi30K and FLORES-200.