Several recent systems learn a visual or audio task without ever seeing an image or a sound. They train a small model on top of a frozen text encoder from a contrastive model, meaning a model like CLIP that was trained to place matching images and captions near each other. At test time they swap in the matching image encoder. Separately, multilingual models fine-tuned only on English often work in other languages they were never fine-tuned on. This note asks whether one model can do both jumps at once. The model would be trained only on English text. At test time it would receive an image together with a question or hypothesis written in, say, Arabic or Korean, and it would have to answer correctly. It would have seen no images, no target-language task data, and no machine-translated training data.
The two jumps fail for related reasons. Image embeddings sit in a region displaced from text embeddings, a property called the modality gap. Embeddings of non-English text sit in displaced "language islands" relative to English. A model trained on English text embeddings faces both displacements at once. The working hypothesis is that, inside a frozen multilingual contrastive encoder, each displacement is mostly a low-rank shift. Each can be estimated from unpaired data alone (a pile of images, a pile of monolingual sentences), and the two roughly add. If that holds, we can train the task model on English embeddings that are deliberately pushed along the estimated displacement directions. This would replace the uniform random noise that current text-only methods add to embeddings, which may wash out the fine distinctions that cross-lingual transfer depends on.
Core Claim
Text-only visual reasoning can be made to transfer jointly across modality and language by shaping training-time perturbations to the measured geometry of the two gaps, without adding noise of the same size in every direction. The claim has two parts, and either can fail on its own. The first is a geometric claim. The image-minus-English-text displacement and the language-minus-English displacement occupy nearly separate low-dimensional subspaces, and their effects on a downstream model compose approximately additively. The second is a method claim. Training on "gap-shaped" augmentations estimated only from unpaired statistics beats the best uniform-noise recipe on image-grounded reasoning in unseen languages. The gain should be largest for languages far from English.
This differs from the closest work in what it assumes is available. MultiCapCLIP trains a captioner without image-caption pairs. It does so by auto-encoding sentences in each target language through a fixed dictionary of 1,000 visual concepts, so it needs target-language text for the task and is limited to captioning. ZeroNLG builds its shared space from 400M image-English pairs plus machine-translated CC3M. It is not training without images or translations, and it also targets generation only. Scheduled Unfreezing improves English-to-other-language transfer for vision-language encoders, but it fine-tunes on paired English image-text data. LRM-LLaVA relies on over 120M image-text pairs and instructions translated by machine translation. None of these trains a reasoning model on English text alone and evaluates it on images paired with non-English queries. None asks how the two gaps interact geometrically. That second question is listed as open in the source material, alongside the specific worry that uniform noise may erase cross-lingual distinctions.
Research Program
The setup uses a frozen image encoder and a frozen multilingual text encoder that live in a shared contrastive space. The multilingual text encoder is a publicly available multilingual CLIP-style encoder. Its pretraining used parallel text, which matches the usual standing of mBERT-like models in cross-lingual transfer. The important point is that no task data in any target language and no images touch the trainable parameters. On top of the frozen encoders sits a small task model: a cross-attention classifier for entailment and an answer classifier for question answering.
Training follows the text-only recipe of CLOSE. That paper trained T5 on CLIP text embeddings plus Gaussian noise and transferred to SNLI-VE, VQA, and captioning. In CLOSE, a caption stands in for the image. For entailment, the English SNLI premise is a caption, and we pair it with the English hypothesis and label. For question answering, we use English captions with question-answer pairs, as CLOSE did with VQA-E. At test time the premise embedding is replaced by the image embedding. The hypothesis or question is encoded from target-language text by the same multilingual encoder.
The new ingredient is how training embeddings are perturbed. From unpaired data we estimate two things: the mean and covariance of image embeddings, using unlabeled images, and the mean and covariance of each language's text embeddings, using monolingual sentences. Let $z$ be an English text embedding, and let $\mu_s,\Sigma_s$ and $\mu_t,\Sigma_t$ be the mean and covariance of the source (English text) and a target region (images, or language $\ell$). The moment-matched displacement of $z$ toward the target region is
$$ T_t(z) = \mu_t + \Sigma_t^{1/2}\,\Sigma_s^{-1/2}\,(z-\mu_s), $$
truncated to the top $r$ eigen-directions where $\Sigma_s$ and $\Sigma_t$ differ most. The truncation rank $r$ controls how much shape, beyond a pure mean shift, the correction carries. During training, the premise embedding is moved toward the image region. The hypothesis or question embedding is moved toward a randomly drawn language from a pool of languages with available monolingual text. In both cases the movement is a random interpolation between $z$ and $T_t(z)$, plus small residual noise. The task model therefore sees inputs in the regions it will meet at test time, without the noise being spread in every direction.
The geometric half of the program is a diagnostic study. It uses held-out paired data for analysis only, never for training. On image-caption pairs and their translations, we measure per-example displacements. Three quantities are of interest: the principal angles between the modality-displacement subspace and each language-displacement subspace; how much of the image-to-Arabic displacement is explained by the sum of the image-to-English and English-to-Arabic displacements; and whether the unpaired moment estimates recover the paired displacement directions. We also test the concern from the modality-gap literature directly. We measure how much uniform noise, at the variance CLOSE uses, shrinks the margin between cross-lingual nearest neighbors, broken down by language. ModGap reports that the image-text gap in standard CLIP is large (distance 0.47) yet can be closed to 0.03 without hurting retrieval. That result suggests the modality gap is largely a removable offset. The diagnostic checks whether language islands have the same simple structure inside a multilingual CLIP-style space.
Baselines
The naive reference point is direct substitution with no regularization. The task model is trained on clean English text embeddings, and image and target-language embeddings are fed in at test time. The source material reports that this collapses in the modality-only case. One example is the unaugmented captioning baseline in MultiCapCLIP, which scores 2.8 BLEU@4 on Chinese VATEX against 8.4 with its augmentation. The naive baseline anchors how much any correction buys.
The strongest existing competitor for this exact setting is CLOSE-style uniform Gaussian noise injection, ported to the multilingual encoder. It is the only established recipe that trains reasoning models with zero images. The same idea, applied to audio, brings text-only captioning within reach of paired training, with AudioCaps SPIDEr of 0.437 versus 0.455 (Training Audio Captioning Models without Audio). For fairness, the noise variance gets the same tuning budget as our rank $r$ and interpolation range. Tuning in both cases uses a validation split made from English text plus a small set of held-out unlabeled images. No target-language labels are used for model selection, because selecting on target-language labels would quietly break the zero-shot claim. No reported result was found on Lacuna for text-only training evaluated on XVNLI or xGQA, so the noise baseline must be run, not quoted.
A second strong competitor is the mean-shift correction: subtract the estimated image-minus-text and language-minus-English mean offsets at test time. It is the rank-zero special case of our method, and it isolates whether the covariance shaping matters at all. The key ablations each remove one piece. One applies only the modality displacement. One applies only the language displacement. One sweeps $r$ from zero upward. One replaces our language-aware perturbations with uniform noise matched in total variance, which isolates direction from magnitude.
For calibration we report models that use paired data, but we do not count them as competitors. IGLUE reports the best model at roughly 77% accuracy on English XVNLI and about 62% in other languages (IGLUE). Scheduled Unfreezing reports 41.3 average zero-shot xGQA accuracy with UC2, up from 31.9. Our method beats the field if it exceeds both uniform noise and mean-shift on average non-English accuracy across XVNLI and xGQA, with a larger margin on languages far from English. If it also approaches the image-trained transfer numbers, that would be notable, but it is not required.
Why not just…?
Why not just translate the target-language question into English at test time? This is the most serious threat. IGLUE found that translating test inputs into English often beats native multilingual vision-language models. Our method can be combined with this approach, so the comparison is run cleanly. The same English-text-only model receives machine-translated hypotheses and questions from a fixed open translation system, and we report where each approach wins. We expect translation to win on high-resource languages. Gap-shaped training is likely to compete only where translation is weak or ambiguous, given VATEX's evidence that text-only translation garbles polysemous phrases without visual context (VATEX) and the finding that reading-comprehension transfer degraded under machine translation (Zero-shot Reading Comprehension). If translation wins everywhere, the method claim fails. The geometric findings still stand, and we would also report how much translation and gap shaping add when used together.
Why not just train on machine-translated English captions and questions? This is what ZeroNLG and LRM-LLaVA do. LRM-LLaVA itself names translation quality and translationese as a bottleneck for low-resource content, and Multi30K shows that translated text mirrors source syntax (Multi30K). Still, it is a fair upper reference. We train the same task model on translated SNLI and caption-question data, using the same translation system, and report it as "translate-train." If gap shaping, which uses only monolingual statistics, closes most of the distance to translate-train, then it is useful exactly where parallel resources are thin.
Why not just increase the Gaussian noise until it covers both gaps? This is the cheapest alternative, and it is what the uniform-noise baseline sweep tests. Our hypothesis predicts that large uniform noise will help high-resource languages but hurt distant ones, by blurring the margins between cross-lingual neighbors that the diagnostic measures. If the best uniform noise matches our method across all languages, we drop the method claim and report that the gaps are effectively isotropic at the scale that matters.
Why not just subtract the mean offsets? If mean-shift matches the full method, the honest conclusion is that both gaps are first-order shifts, and the recommended recipe becomes the simpler one. The rank sweep makes this outcome visible rather than hidden.
Proposed Experiments
The primary evaluation is IGLUE's XVNLI and xGQA in all their target languages, with MaRVL as a stress test of culturally distant images. The metric is accuracy, reported per language and as the non-English average, alongside English to show the cost of the modality jump alone. Training uses only English SNLI and English caption-question-answer text. Unpaired statistics come from a disjoint set of unlabeled images and from monolingual text per language, with 10k, 100k, and 1M sentences to test how many samples the estimates need. A secondary generation check uses Chinese VATEX and Multi30K German. A multilingual decoder is trained on English captions only, with gap-shaped perturbations, and compared against MultiCapCLIP's reported 8.4 BLEU@4 on Chinese VATEX. That paper used Chinese text in training and we do not, so a smaller gap to its score already says something.
Three checks decide the outcome. The first: if the measured modality and language displacement subspaces are not nearly orthogonal, or their sum explains little of the combined displacement, the composition hypothesis is false and per-gap estimation has no basis. The second: if gap-shaped training fails to beat the best-tuned uniform noise on the non-English average, the method claim fails. The third: if the improvement over uniform noise does not grow with typological distance from English, the proposed mechanism (uniform noise erasing fragile cross-lingual distinctions) is not what drives the gain. A fourth, smaller check repeats the composition test with a frozen audio-text encoder on Clotho using translated test captions. Training without audio reports 0.256 SPIDEr in English there, so the check would show whether the geometry generalizes beyond vision.
References
- MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning
- ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation
- Improving Vision-Language Cross-Lingual Transfer with Scheduled Unfreezing
- LRM-LLaVA: Overcoming the Modality Gap of Multilingual Large Language-Vision Model for Low-Resource Languages
- I can't believe there's no images! : Learning Visual Tasks Using Only Language Supervision
- Closing the Modality Gap Aligns Group-Wise Semantics
- Training Audio Captioning Models without Audio
- IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
- VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
- Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model
- Multi30K: Multilingual English-German Image Descriptions