Fairness Debt Inside the Transport Plan: Online Equalization of Cross-Lingual Visual Grounding

Multilingual vision-language models are meant to use images as a shared anchor across languages. A photo of a dog on a beach is the same photo whether it is captioned in English or Swahili, so text in every language should land near the same visual features. In practice this does not happen. Non-English queries lose much of their accuracy, and translating them into English at test time often beats the multilingual model outright. We propose to treat the gap between languages as a running debt that the training procedure must pay back, and to pay it back in one specific place: the matching between caption words and image regions.

The mechanism combines two existing ingredients. The first is optimal transport (OT), a way of matching two sets of items so that every unit of "mass" on one side is assigned somewhere on the other, at minimum total cost. The second is an online fairness auditor. This is a learner that keeps a per-language tally of how far that language's grounding error sits above the average. Languages that fall behind get a larger share of the image's region mass in a joint word–region matching. Their words are then forced to compete with English words for the salient regions, instead of being matched to leftovers. The claim is that this reallocation does two things at once. It pulls target-language tokens toward the same visual anchors as English, which addresses representation misalignment directly. And it bounds the cumulative disparity across languages so that it grows sub-linearly over the training stream.

Core Claim

The fixed fairness targets in fair optimal transport should instead be set online by a no-regret dual player driven by per-language grounding error. Placing those targets in the column marginals of a joint multilingual word–region transport problem aligns languages better than reweighting their losses. Concretely, take an image $i$ with region features and captions in a set of languages $L$. We solve one entropy-regularized, partially unbalanced transport problem. The rows are the image's regions. The columns are the union of all caption tokens across languages. The total column mass given to language $l$ is

$$w_l(\alpha_t) = \frac{\exp(\alpha_{t,l})}{\sum_{k \in L} \exp(\alpha_{t,k})},$$

where $\alpha_{t,l} \ge 0$ is language $l$'s fairness debt at step $t$. The debt is updated by projected online gradient descent on the violation

$$g_{t,l} = \ell_{t,l} - \bar{\ell}_t - \delta.$$

Here $\ell_{t,l}$ is a held-out grounding loss for language $l$ (image–text retrieval loss on a small rolling validation buffer), $\bar{\ell}_t$ is its mean over languages, and $\delta$ is a tolerated slack. The quantity we want to control is the cumulative violation $V_T = \sum_{t \le T} \max_l [g_{t,l}]_+$. For the dual player alone, standard online linear optimization gives $V_T = O(\sqrt{T})$ against any fixed comparator. Whether the non-convex network reaches the corresponding saddle point is an empirical question, and we test it rather than assume it.

Where This Sits in Prior Work

Three lines of work motivate the gap. The IGLUE benchmark ( IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages) documents large zero-shot drops from English and shows that translate-test often wins. That result is the bar this proposal has to clear. Probing work by Müller et al. finds that alignment degrades in upper layers for distant language pairs, which tells us where to measure. Work on frozen visual backbones, such as Wukong, shows that the visual space can be held fixed while text encoders are mapped into it. We adopt that setting so that images really are a fixed anchor.

Three lines of work supply the method. UNITER uses OT for word–region alignment, but it matches one caption at a time with fixed uniform marginals. OTSeg shows that Sinkhorn normalization makes multiple prompts compete for pixels instead of all hijacking the salient region. We carry that competition over to languages. Optimal Transport under Group Fairness Constraints builds group proportions into Sinkhorn (FairSinkhorn), but it lists the exogenous, static target matrix as a limitation. Making that target adaptive is the gap our proposal fills.

The online side reuses the primal-dual "fairness debt" structure from Online Combinatorial Optimization with Group Fairness Constraints and FairDolce. Both of those papers bound regret under convexity assumptions. Neither acts on representation alignment. We inherit their guarantee only for the dual update and state this plainly.

Two further papers are close in spirit and are the ones to beat on alignment. UC2 uses machine-translated captions and co-masking so that the image serves as a pivot. CCLM treats cross-lingual and cross-modal alignment as one contrastive task. Both give languages fixed, equal treatment at the level of the objective. Neither adapts over time to which language is currently behind.

Research Program

We freeze a CLIP-style visual encoder and fine-tune a multilingual text encoder with two losses. The first is a global contrastive loss. The second is the joint multilingual word–region OT loss, which is the transport cost under the debt-weighted marginals.

Training data is a stream of English image–caption pairs plus machine-translated captions, as in UC2. The stream is deliberately imbalanced: languages arrive at realistic, skewed rates, and some appear late. This makes the online, sub-linear framing necessary rather than decorative. A fixed reweighting tuned on the final mixture would not see the order in which languages arrive, and would not respond when a language suddenly falls behind.

Unbalanced transport, meaning a soft penalty on unmatched mass rather than exact conservation, lets background regions stay unmatched. This avoids the forced-matching failure that the OT literature flags. We analyze three things: the dual variables' trajectories, the empirical growth of $V_T$ against $\sqrt{T}$ and $T$, and layer-wise cross-lingual distances between translation pairs.

Baselines

The naive reference is the same frozen-vision model fine-tuned on English only and evaluated zero-shot. It shows how large the gap is before anything is done.

The strongest practical competitor is translate-test: target queries are machine-translated into English and fed to the English model. IGLUE ( IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages) reports that it often beats multilingual models. However, no reported result found on Lacuna for translate-test or for UC2/CCLM on IGLUE tasks, so we will measure every number ourselves with the same machine-translation system, the same visual encoder, and the same data. Beating translate-test on average across languages, or matching it at lower inference cost while beating it on MaRVL (where translation may strip cultural terms), is the headline success criterion.

The strongest training-time competitor is data-level balancing: temperature-based language sampling on the same stream, with the temperature tuned on the same validation buffer. We also run a Euclidean alignment penalty on translation pairs following Cao, and a per-caption OT loss in the style of UNITER with uniform marginals. All methods get the same text-encoder initialization, token budget, and hyperparameter search budget (the same number of trials).

Two ablations isolate the key ingredient. The first applies the identical dual variables $\alpha_t$ as loss weights instead of as transport marginals. This is a group-DRO-style reweighting, and it separates "online debt" from "debt placed inside the matching." The second is static FairSinkhorn with uniform target proportions ( Optimal Transport under Group Fairness Constraints), which separates "adaptive targets" from "fair targets." The proposal succeeds only if it beats both ablations on worst-language accuracy and on upper-layer alignment.

Why not just…?

Why not just translate at test time? It is a real threat, and IGLUE's finding is the reason this proposal exists. Translate-test has three costs. It adds a translation call to every query. It depends on machine translation quality, which is weakest exactly for the low-resource languages we care about. And it cannot help with culturally specific concepts that have no English visual counterpart. We report it per language. If it dominates everywhere, including MaRVL, the proposal fails, and we will say so.

Why not just reweight each language's loss with the same fairness debt? This is the most direct simpler alternative, and it is built in as the first ablation. Our hypothesis is specific. Loss reweighting scales a lagging language's gradient uniformly. Marginal reallocation in a joint transport problem changes which regions that language's tokens are matched to, and so moves them onto the same anchors as English. If the reweighting ablation matches our method on both accuracy and layer-wise alignment, the transport placement adds nothing, and the contribution reduces to online group reweighting.

Why not just align translation pairs directly, or use code-switching? Direct alignment (Cao) and dictionary code-switching (CoSDA-ML) need parallel text or dictionaries. They also treat all languages equally regardless of how far behind each one is. Dictionary substitution is also known to break agreement and to mishandle words with several meanings. We compare against direct alignment under the same data. The image anchor should matter most for pairs whose translations are noisy.

Proposed Experiments

We evaluate on IGLUE's tasks: multilingual image–text retrieval, cross-lingual visual question answering, visual entailment, and MaRVL's culturally diverse reasoning. The reported metrics are mean accuracy or recall, worst-language score, and the max–mean gap.

The decisive checks are three. First, the empirical cumulative violation $V_T$ should grow roughly as $\sqrt{T}$ under our method. Under temperature sampling it should grow linearly when a late-arriving language joins the stream. Second, cosine distance between translation pairs in the top text-encoder layers, measured following Müller et al., should shrink more under joint transport than under either ablation. Third, worst-language retrieval should improve without lowering English by more than the tolerance $\delta$.

A negative result on the second check, together with a positive result on the first, would show that online debt equalizes scores without actually aligning representations. That outcome would still be worth reporting, because it separates two effects that current multilingual training conflates.