Does Scale Wash Out Verb Noise Across Languages?

Multilingual vision-language models learn to match images with text in many languages. They are usually trained on English web captions that have been machine-translated into other languages. A common working belief is that noise in such data dilutes with scale: wrong labels scatter at random, so the true signal wins as the corpus grows. We argue that this belief holds for nouns but fails for verbs. When a translation system meets an ambiguous English verb, it does not err at random. It picks the most frequent target-language sense almost every time. The English verb blow becomes one German verb whether the picture shows someone inflating a balloon or a bomb detonating. That error is systematic and repeats identically across millions of captions. More data should therefore reinforce it rather than dilute it. If this is right, the noun-versus-verb performance gap in non-English languages will stop shrinking as translated data grows, even while the English gap and the target-language noun performance keep improving.

The proposal has two parts. The first is a controlled measurement: part-of-speech-resolved scaling curves for zero-shot cross-lingual transfer, meaning training with English task labels and testing in other languages. The curves test whether verb errors behave like random noise or systematic noise. The second is a targeted repair: image-conditioned re-selection of target-language verbs in translated captions. The rejected verb senses are then reused as hard negatives, that is, wrong captions that differ from the right one in a single word. The contribution is the diagnosis and the fact that it predicts which fix works. Generic noise-robust training and generic caption filtering should not close the gap, and the experiments are designed to show whether that is true.

Core Claim

Treat per-category retrieval or matching error in a target language as a function of translated training data size $n$. We model it as

$$\mathrm{err}_c(n) \approx a_c\, n^{-b_c} + e_c ,$$

where $c$ is a word category (noun or verb), $a_c$ and $b_c$ describe how fast error falls, and $e_c$ is a floor set by noise that is correlated across examples. The functional form is the usual power-law-plus-constant used in data scaling work such as How Much is a Noisy Image Worth? Data Scaling Laws for Ambient Diffusion.. That paper measures how noisy samples trade against clean ones and finds the clean/noisy gap narrows with volume. That narrowing is exactly what we predict fails for verbs.

We make three predictions. First, $e_{\text{verb}} > e_{\text{noun}}$ in target languages but not in English. Second, $e_{\text{verb}}$ tracks the translation system's verb-sense error rate. Third, replacing systematic verb corruption with random corruption at the same rate lowers $e_{\text{verb}}$ back toward $e_{\text{noun}}$. The third prediction is the decisive one. It separates "verbs are just harder" from "verb noise is the wrong kind of noise." It also turns the observation in DrugOOD into a testable claim. That observation is that scaling noisy data helps only until the noise becomes systematic.

Why This Gap Is Open

Three groups of prior work supply the pieces, but none connects them.

The first group establishes the verb weakness, and all of it is English-only. Going Beyond Nouns With Vision & Language Models Using Synthetic Data and When and Why Vision-Language Models Behave like Bags-Of-Words show that contrastive image-text models match captions mostly by object nouns. They repair this with synthetic action data or with rule-based English hard negatives, respectively. Neither tests other languages. The Bags-Of-Words paper states that building valid hard negatives in typologically diverse languages is open.

The second group handles translation noise but treats all words alike. UC2 pretrains on machine-translated Conceptual Captions. Its region-to-token objective binds image regions to object nouns and does not bind actions to verbs. NRCCR and CL2CM down-weight or re-align noisy translated tokens by checking them against the English source. That design has a blind spot we exploit. A fluent wrong-sense verb is aligned to its English source verb, so a source-conditioned noise detector sees nothing wrong. Only the image reveals the error.

The third group shows the verb-translation problem exists but has not scaled it. Seeing the Action builds MultiSense, which covers 55 English verbs with German and Spanish translations chosen by image. It shows that a plain translation system gets the verb right only 22.9% of the time, rising to 49.5% with visually predicted verb constraints. That paper motivates our claim that translation verb errors are frequent and image-resolvable. Its scope is limited: 55 curated verbs, no pretraining, and no scaling. Cross-Lingual Induction and Transfer of Verb Classes constrains our design. It finds that verb class structure is distinct from surface similarity, which warns us that verb meaning will not fall out of generic similarity objectives.

Finally, Improving Vision-Language Cross-Lingual Transfer with Scheduled Unfreezing shows that much of the transfer loss comes from English fine-tuning damaging lower layers. That is an optimization explanation competing with our data explanation, and we test it directly.

Research Program

Stage one: build a part-of-speech-resolved evaluation in target languages. No such evaluation exists in the file. We start from the test captions of xFlickr&CO and Multi30K. For each caption we create two minimal-pair distractors in the target language. One swaps a concrete noun and the other swaps the main verb, and each is checked by a native speaker for fluency and for truly changing the meaning. Verb swaps are drawn preferentially from alternative senses of the same English verb, which targets the confusion we hypothesize. This yields noun-swap and verb-swap image-to-text accuracy per language, in the style of ARO but multilingual.

We add two further measures. MultiSense gives verb-sense selection in German and Spanish. For English reference points we use ARO and Winoground.

Stage two: scaling curves under three noise regimes. We take a fixed multilingual image-text encoder initialization and continue training on Conceptual Captions subsets of 0.3M, 1M, and 3M pairs. The initialization is UC2's released model if available, otherwise XLM-R text paired with a CLIP image tower. The training text comes in three versions.

  • Natural translation: the captions as machine-translated.
  • Oracle-free random corruption: target verbs replaced by a random other sense of the same English verb, at a rate matched to the measured translation verb-error rate.
  • Systematic corruption: target verbs always replaced by the single most frequent sense.

We measure the translation verb-error rate on a held-out sample annotated against MultiSense-style sense inventories. Fitting the equation above per category and per regime gives $e_c$ with bootstrap intervals.

Stage three: the repair, image-conditioned verb re-selection. For each translated caption we locate the target word aligned to the English main verb. We then generate candidate target verbs from the translation system's n-best list and a bilingual dictionary. Each candidate is back-translated to a short English sense gloss. The gloss is scored against the image by an English model that is already relatively verb-aware (NegCLIP or syn-CLIP).

Routing the sense check through English avoids a circular dependency. We do not need a target-language model to already understand verbs in order to teach it verbs. This is the text-to-text scoring idea of Sieve, applied to a single word rather than a whole caption.

The top candidate replaces the translated verb. The losing candidates become sense-contrastive hard negatives in a contrastive loss with in-batch negatives. We then re-run the stage-two scaling curves with repaired data.

Baselines

Naive reference. The do-nothing baseline is continued training on unrepaired translated captions at each data size, with standard contrastive loss and full fine-tuning. This defines the scaling curve the proposal claims has a verb floor. As an external anchor, UC2 reports a Multi30K meta-average recall of 86.2 against M3P's 82.8 (UC2). No part-of-speech-resolved number exists on Lacuna for any multilingual model, so our split measures will be new.

Generic noise-robust training. This baseline is NRCCR-style source-conditioned token down-weighting with self-distillation (NRCCR). It is the strongest existing method aimed specifically at translation noise, and it reports 81.28 mAP against 61.84 for the basic model on VATEX cross-lingual text-to-text retrieval. We reimplement its filtering and distillation losses on our image-text setup, using the same data, the same number of steps, and a matched hyperparameter search of equal trials per method. Beating it means lower target-language verb-swap error at every data size with no loss in noun-swap accuracy or overall retrieval recall.

English hard negatives. This baseline is NegCLIP-style English-only hard-negative fine-tuning (Bags-Of-Words), evaluated zero-shot in target languages. It tests whether verb competence transfers across languages without target-language data. For reference, NegCLIP raises English VG-Relation accuracy from 63% to 81%. We run it both with standard fine-tuning and with Scheduled Unfreezing (Scheduled Unfreezing). That paper reports UC2 on xGQA rising from 31.9 to 41.3 with the schedule, so the optimization explanation gets its best shot.

Translated hard negatives (the key ablation). This baseline takes NegCLIP's English verb-swapped negatives and machine-translates them into each target language. It isolates the proposal's key ingredient, which is that negatives and positives must differ in sense after translation. When the translation system collapses senses, a translated negative can become identical to the positive. Comparing this against our sense-contrastive negatives measures exactly that effect.

Repair ablation. We also run verb re-selection without hard negatives, and hard negatives drawn from random rather than image-rejected senses. This separates cleaning the positives from contrasting the senses.

Why not just…?

Why not just scale the translated data further? This is the null hypothesis the scaling curves test. The paper's claim is that the verb floor $e_{\text{verb}}$ is set by systematic error and does not move with $n$. If the natural-translation verb curve keeps falling at the same exponent as nouns through 3M pairs, the claim is wrong and we will say so. The random-versus-systematic corruption comparison settles it independently of how far we can scale.

Why not just filter harder with a captioner-and-filter pipeline? BLIP's caption filtering and Sieve both score whole-caption agreement with the image. The scorers they rely on are the bag-of-objects models documented in Going Beyond Nouns. A caption with the right nouns and a wrong-sense verb therefore passes. We run Sieve-style whole-caption filtering at matched retained-data size as an extra baseline. We expect it to improve noun-swap and overall recall but leave verb-swap nearly unchanged. If it closes the verb gap, our word-level repair is unnecessary.

Why not just translate the test query into English and use an English verb-aware model? This is a real threat, because it sidesteps training noise entirely. It meets the same translation sense errors at test time, however. Also, Zero-shot Reading Comprehension by Cross-lingual Transfer Learning found that explicit test-time translation often degraded performance compared with shared multilingual representations. We include translate-test with NegCLIP as a full baseline on the verb-swap benchmark. On verb-swap items built from sense alternatives, it should fail wherever the translation system maps both senses to one English verb. If it matches our method anyway, the practical conclusion is to translate at test time, and we would report that.

Why not just fix fine-tuning with Scheduled Unfreezing? The schedule changes how English task data damages lower layers and does not touch pretraining data. If the transfer gap were purely optimization, it would close noun and verb gaps alike. Our prediction is that it helps both categories roughly equally and leaves the verb-minus-noun difference intact. The NegCLIP-plus-schedule baseline measures this directly.

Proposed Experiments

The experiments cover German, Spanish, French, and Czech through Multi30K and xFlickr&CO, which give many languages. They also cover Chinese, Japanese, and further IGLUE languages through xFlickr&CO, giving typological distance. German and Spanish receive additional MultiSense evaluation.

The metrics are noun-swap and verb-swap minimal-pair accuracy per language, MultiSense verb accuracy, and standard recall on xFlickr&CO and Multi30K. We also report XVNLI and xGQA accuracy to check that repairs do not harm task transfer. Exact IGLUE per-language figures for these models were not available on Lacuna beyond the xGQA numbers above, so we will rerun all baselines ourselves.

The decisive checks are four.

  1. Is the verb floor real? The fitted verb floor under natural translation should exceed the noun floor in target languages but not in English.
  2. Does noise type matter? Random verb corruption at matched rate should produce a lower verb floor than systematic corruption. This is the direct test that noise type, not verb difficulty, sets the floor.
  3. Does the repair beat the baselines where it should? Sense-contrastive repair should lower the verb floor more than NRCCR-style training, whole-caption filtering, and translated NegCLIP negatives. The gain should be largest on MultiSense's ambiguous verbs.
  4. Is the repair free? Noun-swap accuracy and overall recall should stay within bootstrap noise of the baseline.

A positive result would show that "scale dilutes noise" is category-dependent, and that cross-lingual verb competence needs sense-level correction rather than more data. A negative result on check two would show that verbs are simply harder in every language. That finding is also informative, because it would redirect effort toward model design rather than data.