Grounding Phrases Across Languages When the Camera Also Changes

Visual grounding means finding the box in an image that a phrase refers to, such as "the man in the red jacket." Systems trained on English phrases and ordinary web photos fail in two ways when deployed. The query may come in another language, and the images may come from a different camera, lighting condition, or rendering pipeline. Each problem has its own literature. Domain adaptation fixes image shift by changing pixels (restyling source images to look like target images) and by aligning internal features. Cross-lingual grounding transfers an English grounder to French or Japanese while holding the images fixed. No work in our search tests what happens when both shifts occur together. No work asks whether the two literatures' tools can simply be stacked.

This note makes two claims. The first is an empirical test: are language shift and image shift additive in grounding error, or do they compound? Our hypothesis is that a low-resource language grounder relies on fewer, more fragile visual cues, so it breaks disproportionately under image shift. The second is a method that follows if they compound. It adapts images at both the pixel and feature level, using a grounding-specific consistency check instead of a class label: after restyling, the English and Japanese versions of the same phrase must still point to the same box. Parallel phrases in two languages also act as a filter. Consistency is enforced only where the two languages' grounders agree on the original image. This targets the known weakness that pixel translators distort the very attributes ("blue," "small," "behind") that referring phrases mention.

Core Claim

Our hypothesis is that the error from a new language and the error from a new visual domain interact positively. If so, adapting the visual encoder with a phrase-level, multilingual consistency constraint closes more of the compound gap than either of two alternatives. One is the obvious cascade: machine-translate the query to English, then run an English grounder adapted with standard visual domain adaptation. The other is either adaptation level alone. The claim is falsifiable. If the interaction term is near zero, the cascade is the right answer, and we would report that.

We define the interaction as follows. Let $E(\ell, d)$ be grounding error for query language $\ell \in \{\text{en}, \text{tgt}\}$ on image domain $d \in \{\text{src}, \text{tgt}\}$. The interaction is

$$ I = E(\text{tgt},\text{tgt}) - E(\text{tgt},\text{src}) - E(\text{en},\text{tgt}) + E(\text{en},\text{src}). $$

$I > 0$ means the two shifts compound. $I \approx 0$ means they are separable, and independent fixes for each can be composed.

Relation to Prior Work

Two adjacent works motivate the gap.

Cross-Lingual Visual Grounding transfers an English referring-expression model to French. It reaches 65.17% accuracy against 69.04% for English, but the French and English data share the same web-image distribution. We add a simultaneous image shift and ask whether its feature-level transfer survives it.

CLEAR handles multilingual instructions and unseen buildings in navigation. It makes visual features ignore room aesthetics through object matching on Mask R-CNN outputs. It does nothing in pixel space and does not evaluate box-level grounding. We work at the box level and treat pixel-level adaptation as a first-class component.

CultureCLIP adapts CLIP with synthetic culturally contrasting images. It operates on whole images and never uses unlabeled real target imagery. We use only unlabeled real target images.

The multi-level adaptation literature supplies the method.

CyCADA couples cycle-consistent pixel translation, a semantic-consistency loss from a fixed source classifier, and feature-level adversarial alignment. Its ablations show that the levels are complementary. Its consistency loss assumes a closed label set. We replace it with a phrase-to-box consistency loss that works for open-vocabulary phrases in several languages.

Gotta Adapt 'Em All separates nameable photometric and geometric shifts from residual feature shift. On CompCars it reaches 84.20% against 64.82% for CyCADA, which supports explicitly modeling nameable factors.

I2F and DCAN show that non-adversarial image-level normalization plus feature-statistics alignment is competitive and stable. I2F reports 58.2% mIoU and DCAN 41.7% on GTA5→Cityscapes. We adopt their non-adversarial feature alignment because it avoids GAN instability on a grounding backbone.

Constraining evidence comes from two papers.

IGLUE reports that translate-test often beats multilingual multimodal encoders, and that few-shot multimodal learning curves are flat. The first result makes the cascade a serious competitor rather than a strawman. The second argues against "just annotate a few target examples," but we test that directly.

xGQA attributes part of the cross-lingual cliff to English-trained region detectors. This shapes our diagnostics: we measure proposal recall separately from phrase-to-proposal scoring.

Research Program

Benchmark construction. No benchmark in our search pairs non-English referring phrases with a shifted image domain, so we build one in two tiers.

The controlled tier uses Flickr30k Entities JP. It has parallel English and Japanese phrases linked to the same boxes, with reported Recall@1 of 30.2% for Japanese and 32.3% for English. We apply parameterized shifts to the test images, keeping boxes and parallel phrases intact:

  • low light with sensor noise,
  • color-profile change,
  • downsampling with compression,
  • a learned painterly or synthetic-render style.

Because the boxes and phrases are identical across conditions, all four cells of $I$ can be measured exactly.

The natural tier uses unlabeled Cityscapes training images for adaptation. A test set of roughly 1,500 referring phrases on Cityscapes validation images will be written natively in English, Japanese, and French, not translated. We also include the French grounding images from Dong et al. to cover a second language under the controlled shifts. The source domain is always Flickr-style web photos with annotations. Target-domain images carry no annotation in any language.

Model. The grounder is two-stage. A region proposer feeds a phrase-to-region scorer, and the phrase side uses a multilingual encoder (XLM-R, as in AltCLIP). We first train it on English and Japanese source data.

Adaptation. We add three components:

  1. Image-level stage. Explicit photometric and texture normalization in the style of I2F handles nameable shifts. A light learned translator handles the residual style.
  2. Feature-level stage. Channel statistics of region features are aligned to target statistics, as in DCAN, with no discriminator.
  3. Grounding-consistency loss. For each source image and each parallel phrase pair, the scorer's distribution over proposals on the restyled image must match its distribution on the original. This is enforced for both languages jointly, and only on phrase pairs where the English and Japanese grounders agreed on the original image.

The filter in the third component addresses CyCADA's dependence on a reliable source model. It is the specific place where the cross-lingual signal does work that a monolingual pipeline cannot.

Diagnostics. We report proposal recall under shift, separately from scoring accuracy given the gold proposal. This tells us whether the compound failure lives in the English-trained detector or in the language-conditioned scorer.

Baselines

Source-only (the do-nothing reference). The grounder is trained on English and Japanese source data and evaluated directly on shifted images. Every method is reported as the fraction of the source-only gap to the in-domain result that it recovers. For orientation on in-domain cross-lingual grounding, the file reports 30.2% Japanese and 32.3% English Recall@1 on F30kEnt-JP with GCCA, and 65.17% French against 69.04% English in Dong et al. We found no reported result on Lacuna for any grounding method under simultaneous language and image shift, so all shifted-condition numbers will be new.

Translate-test plus visual adaptation (the strongest competitor). Target-language queries are machine-translated to English. An English-only grounder is adapted to the target domain with the same image-level and feature-level pipeline as ours, but its pixel consistency comes from a CyCADA-style constraint on English phrases only. IGLUE shows that translate-test frequently wins in multilingual vision-language tasks, which makes this the strongest competitor. It uses the identical backbone, unlabeled target images, adaptation steps, and hyperparameter search budget. The translator is a fixed off-the-shelf system, not tuned on our test data. Beating it means higher target-language Recall@1 on both tiers, by more than the seed-to-seed spread across three seeds, with no loss on English.

Cross-lingual transfer without visual adaptation. This is the Dong et al. feature-transfer approach, applied as published to the target language. It tests whether linguistic transfer alone is robust to image shift.

Single-level ablations. We run feature-only alignment, and pixel-only adaptation with no feature stage. Each is trained under the same budget.

Key-ingredient ablations. These isolate the contribution. We compare our phrase-to-box consistency against three variants:

  • a CyCADA-style region-class consistency computed from detector labels,
  • English-only phrase consistency,
  • multilingual consistency without the agreement filter.

If the multilingual, filtered version does not beat English-only consistency on target-language queries, the cross-lingual part of the method is unjustified.

Why not just…?

Why not just translate the query to English and adapt an English grounder? This is the real threat. If $I \approx 0$, it should win, because each shift is solved by its best specialist. The evidence against it is indirect. The F30kEnt-JP authors document that text-only translation mis-resolves references that depend on the image, such as 乗り物 versus 自動車. Dong et al. note that translation errors propagate directly into grounding misses. Neither measures this under image shift. Our four-cell interaction measurement and the head-to-head on natively written Cityscapes phrases settle it. Native phrases matter because translated test phrases would favor the cascade.

Why not just align features, with adapters or a discriminator? In Gotta Adapt 'Em All and CyCADA, feature-only alignment left large residual gaps compared with joint pixel-and-feature adaptation. In xGQA, adapters alone left Japanese and other languages far below English. That evidence is from classification, segmentation, and question answering, not grounding, so our feature-only ablation tests it directly. Our controlled tier also predicts where pixel-level steps should matter most: low-light and sensor-noise shifts, where low layers carry the discrepancy.

Why not just restyle source images with a strong generator and train on them? CyCADA documents label flipping without a consistency constraint. I2F notes that generators hallucinate texture around boundaries. For grounding, the danger is that the generator changes an attribute the phrase mentions, so the training label silently becomes wrong. We will measure this by counting how often the source grounder's chosen box changes after restyling, broken down by phrases with and without color or size words. The pixel-only ablation shows whether this damage matters in practice.

Why not just use a frozen multilingual foundation model, or annotate a few target examples? AltCLIP and MURAL are strong at matching whole images to text, but neither localizes phrases to boxes. A frozen AltCLIP that scores cropped proposals is cheap, so we include it as an extra reference row. For few-shot annotation, IGLUE found flat learning curves. We test this with 50, 200, and 1,000 annotated target-language, target-domain phrases, both with and without our adaptation. The useful outcome is a shift in that curve, not just a better zero-shot point.

Proposed Experiments

Decisive check: the interaction term. We estimate $I$ per shift type and per language (Japanese, French), with bootstrap confidence intervals over phrases. A clearly positive $I$ licenses the method. An $I$ near zero means we report that the shifts are separable and recommend the cascade.

Main comparison. We compare against all baselines on both tiers. The metric is Recall@1 at intersection-over-union of at least 0.5. Proposal recall and gold-proposal accuracy are reported separately.

Mechanism check. Within the controlled tier, we test whether gains concentrate on phrases containing attribute words. Grounding consistency is designed to protect exactly those phrases.

Detector check. We test whether the remaining compound error sits mainly in proposal recall. If so, the English-detector bottleneck flagged by xGQA is the next target, not further phrase-side adaptation.