Images on the web come with much more text than their caption: the paragraph beside them, the section they sit in, the rest of the page. Some of that text describes what the image shows. Much of it is background story that no picture could verify. Vision-language models are trained by pulling images toward their paired text. When they are fed whole pages, they also absorb this unverifiable text. The usual fixes either discard context and keep only short captions, or replace the web text with synthetic captions. Both throw away the entity-rich knowledge that makes web data valuable. A third family, from older distributional semantics, down-weights words by their distance from a target. That idea is sound, but its decay rate has always been a hand-set constant.
We propose to estimate that decay rate from the data, separately for each language, using a natural experiment that Wikipedia supplies for free. The same image often appears in several language editions, each surrounded by independently written text. A statement near the image in Japanese that is echoed near the same image in Finnish or English is very likely about the image. Counting how often corroborated statements occur at each structural distance from the image (same caption, same paragraph, same section, neighbouring section) gives a per-language relevance-by-distance curve. That curve then weights all context in the language, including statements that appear in only one edition. Native-only local facts are therefore kept when they sit close to the image. They are never penalized merely because English lacks them, which is what translation-consistency denoisers effectively do.
Core Claim
Structural distance from an image is a usable, cheap proxy for whether page text grounds to it. The right rate of decay differs across language editions because their page sizes, layouts and writing conventions differ. That rate can be estimated without labels from cross-edition corroboration on shared images. Concretely, let $d(s,v)$ be the distance in the page's section tree between sentence $s$ and image $v$, measured in structural steps (0 for the caption, 1 for the same paragraph, and so on). For language edition $\ell$, we fit
$$p(\text{corroborated} \mid d, \ell) \approx c_\ell \, e^{-d/\lambda_\ell},$$
where "corroborated" means that a cross-lingual sentence encoder finds a close match for $s$ in another edition's context around the same image. The constant $c_\ell$ absorbs how much of that edition is shared at all, and $\lambda_\ell$ is the edition's relevance half-life in structural units. During training, each context sentence is then weighted by $w(s) = e^{-d(s,v)/\lambda_\ell}$, with a bounded boost when it is corroborated. The weight never falls to zero for lack of corroboration alone. What is new is not distance weighting itself. It is the label-free, per-language calibration of the decay, together with the explicit rule that a missing cross-lingual counterpart is not evidence of noise.
Relation to Prior Work
The distance-weighting literature motivates the mechanism. Neural context embeddings for automatic discovery of word senses found that flat windows over-weight distant generic words. Naive Bayes Word Sense Induction used distance decay to smooth distant context toward uninformative background. Word Sense Disambiguation with Multilingual Features paired a Gaussian window around the target with multilingual evidence. Improving Distributional Similarity with Lessons Learned from Word Embeddings and Redefining Context Windows for Word Embedding Models showed that window and weighting choices can matter as much as the learning algorithm. All of these work on token offsets within sentences and use fixed, hand-tuned decay. We move the idea to section-tree distance on multimodal pages and fit the decay from data per language. Banea and Mihalcea used other languages only as extra features, not as the signal that sets the window.
On the web-noise side, BLIP generates synthetic captions and filters alt-text with an image-grounded scorer. SIEVE prunes pairs by comparing alt-text to a synthetic caption with a sentence encoder. Both operate on isolated image-caption pairs in English. Generalizing Image Captions for Image-Text Parallel Corpus documents why surrounding text is noisy: it references circumstances the image cannot show. That noise is what we try to down-weight rather than delete.
The page-level work both motivates and constrains the design. WikiWeb2M and A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding show that full-page context raises contextual image captioning CIDEr (a caption-overlap score) from 148.53 to 158.19 on English Wikipedia. Their Prefix Global Attention handles distance only through a binary split between global and local tokens. Their result tells us context helps on average. It does not tell us which context helps, and it covers no language other than English. WIT supplies the multilingual material: 37.6M image-text examples over 108 languages, with page and section titles and surrounding text. It treats those fields as a flat bag, and the paper itself points out that the same image carries non-aligned text across editions. That observation is the raw material for our corroboration signal.
The closest cross-lingual competitor is Cross-Lingual Cross-Modal Retrieval With Noise-Robust Fine-Tuning (NRCCR). It uses clean source sentences to filter machine-translated target sentences, enforces round-trip translation consistency, and reports a 19.44 mAP gain over its basic model on VATEX cross-lingual text-to-text retrieval. NRCCR assumes target text should match the source, so any divergence is noise. Our setting reverses that assumption. XOR QA and Cross-Lingual Open-Domain Question Answering with Answer Sentence Generation show that language editions hold complementary, non-overlapping facts; XOR QA reports, for example, a 22% average coverage gain from searching across languages. A denoiser that collapses toward a pivot language would erase exactly those facts. We use corroboration only as positive evidence for calibration.
Research Program
The first step is building the data. Starting from WIT's page URLs, we re-scrape the full section trees for images that appear in at least two language editions, following the WikiWeb2M recipe of keeping section headers, indices, text-only sections and image positions. The goal is a multilingual page-tree corpus covering roughly fifteen to twenty editions. The selection mixes large and small editions and includes the XOR-TyDi languages: Arabic, Bengali, Finnish, Japanese, Korean, Russian and Telugu. For every context sentence we record its structural distance to each image on the page.
The second step is calibration. We embed all context sentences with an off-the-shelf multilingual sentence encoder and mark a sentence corroborated if its nearest neighbour in another edition's context for the same image exceeds a similarity threshold. The threshold is chosen on a small human-labelled set. Before using the fitted curves, we test the exponential form against a free step function over distances, and we fit $\lambda_\ell$ with bootstrap intervals per edition.
The third step is training. A dual-tower image-text model, as in the WIT retrieval setup, receives the caption plus a weighted pooling of context sentences, where the weights are the calibrated $w(s)$. We also train a generative captioner whose page context is weighted in the same way, implemented as an additive bias on attention logits so that it composes with Prefix Global Attention. Images that occur in only one edition receive the decay curve of their language, so the method also applies to the large single-edition majority.
The fourth step is validation. Native speakers label about 3,000 context sentences across six languages as "describes the image or its depicted entity" or "background". About a third of these sentences are deliberately chosen to be native-only, meaning no counterpart exists in any other edition. This set measures whether the weights track grounding. It also checks the main risk of the method: that corroboration mostly finds globally known facts, so native-only local facts might follow a different relevance-by-distance curve.
Baselines
The naive reference point is the WIT setup itself: a dual-tower model trained on reference descriptions alone, and a second version trained on all WIT text fields concatenated without weights. The paper reports Recall@1 of 0.346 on the WIT-All test set for its WIT-trained dual tower, versus 0.048 for a Conceptual Captions model. We rerun that architecture on our page subset rather than quoting the number, because the image set differs.
The strongest existing method for using page context is Prefix Global Attention from A Suite of Generative Tasks, which reaches 158.19 CIDEr on English WikiWeb2M contextual captioning with full-page input. It is the strongest competitor because it already lets a model learn which context to use, at about half the cost of full attention. We train it on the same multilingual page trees, with the same backbone, sequence length, steps and hyperparameter search budget as our model. Its English number serves as a reproduction check before we compare across languages.
For cross-lingual denoising, we adapt NRCCR's cross-attention filtering and cycle-consistency components to filter context sentences, using English as the clean teacher where an English edition exists. No per-language number for this adaptation is reported on Lacuna, and the file gives no Multi30K figures. For per-sentence filtering, we run a SIEVE-style filter and a multilingual image-text similarity threshold at matched retained-token budgets.
The decisive ablations isolate the key ingredient. One uses a single global $\lambda$ tuned on validation data for all languages, which is the classic hand-set decay. A second uses per-language $\lambda_\ell$ tuned by grid search on labelled validation data, which is an oracle upper bound for calibration. A third drops the native-only protection by multiplying weights by corroboration, which is the NRCCR-like assumption.
Beating the strongest baseline means three things. Our model must exceed Prefix Global Attention on non-English contextual captioning CIDEr and on retrieval Recall@1 by more than seed variance. It must do so at matched or lower compute. And it must do so without losing accuracy on native-only facts.
Why not just…?
Why not just let a long-context model's attention learn the weighting? This is the most serious threat. The answer is not settled in advance. Prefix Global Attention already learns its own weighting and gains about 10 CIDEr from page context in English. Our bet is that learned attention needs many examples per language to discover how fast relevance decays, and small editions do not have them. The experiment that settles it plots the gap between our model and Prefix Global Attention against edition size. If that gap is zero for large editions and also zero for small ones, the calibrated prior adds nothing, and we will report that.
Why not just filter each context sentence with image-text similarity or a SIEVE-style captioner? Improved baselines for vision-language pre-training reports that models trained on noisy web data give overconfident, poorly calibrated scores. Captioners also reduce entities to generic words, which is the gap behind WIT's 868,000 versus 150,000 unigram vocabularies. Sentences such as "commissioned in 1806 after Austerlitz" are about the depicted entity but not visual, so a visual-similarity filter should discard them. The human-labelled set measures this directly, by comparing recall of entity-relevant non-visual sentences under each filter at equal retention.
Why not just translate everything into English and use English denoisers? XOR QA's best translate-and-search pipeline reaches only 18.7 average F1 on XOR-Full. NRCCR's own 19.44 mAP gain over a translated basic model shows that translated text is far from clean. Translation also does nothing to identify which native-only sentences are relevant. We include translate-then-SIEVE as a baseline. If it matches us on native-only fact retention, the cross-lingual calibration is unnecessary.
Why not just keep the caption and section title? For many images this may be nearly sufficient. The WikiWeb2M gains show that this approach leaves some signal unused, but those gains are averages. We report results separately for images whose nearest paragraph is empty versus rich, so that any benefit is attributed to the cases where it should appear.
Proposed Experiments and Decisive Checks
The primary evaluations run on our multilingual page-tree corpus. They are WIT-style image-to-text and text-to-image retrieval Recall@1 per language, and contextual image captioning CIDEr per language using held-out native captions as references, with English WikiWeb2M as the reproduction anchor. A third evaluation is a native-only fact probe: for held-out images, we ask entity questions whose answers appear only in a non-English edition's context, and score whether the trained model's retrieval or caption surfaces them. Multi30K and VATEX serve as out-of-domain sanity checks that the weighting does not hurt short-caption retrieval.
Three checks decide whether the idea holds.
The first check asks whether per-language decay is real. If the bootstrap intervals for $\lambda_\ell$ overlap across editions, the per-language claim fails, and the contribution reduces to structural distance weighting with one constant.
The second check asks whether corroboration-calibrated $\lambda_\ell$ comes within a small margin of the oracle labelled-grid $\lambda_\ell$, and beats the single global $\lambda$ on small editions. If it does not, label-free calibration is not doing its job.
The third check asks whether native-only sentences in the human set follow the same relevance-by-distance curve as corroborated ones. If they decay faster or slower, corroboration-based calibration is biased toward globally known content. We would then use the labelled native-only set to estimate a correction factor and report the uncorrected method as limited to globally shared knowledge.
References
- Neural context embeddings for automatic discovery of word senses
- Naive Bayes Word Sense Induction
- Word Sense Disambiguation with Multilingual Features
- Improving Distributional Similarity with Lessons Learned from Word Embeddings
- Redefining Context Windows for Word Embedding Models: An Experimental Study
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- SIEVE: Multimodal Dataset Pruning Using Image Captioning Models
- Generalizing Image Captions for Image-Text Parallel Corpus
- WikiWeb2M: A Page-Level Multimodal Wikipedia Dataset
- A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding
- WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
- Cross-Lingual Cross-Modal Retrieval With Noise-Robust Fine-Tuning
- XOR QA: Cross-lingual Open-Retrieval Question Answering
- Cross-Lingual Open-Domain Question Answering with Answer Sentence Generation
- Improved baselines for vision-language pre-training