Asking About the Words a Model Cannot Ground: Code-Switched Interpretations for Multilingual Interactive Image Search

Vision-language systems that work well in English degrade sharply in other languages. On cross-lingual visual question answering, accuracy falls from about 60% in English to about 20% in the target language (xGQA). On visual inference it falls from about 77% to about 62% (IGLUE). Most work treats this as a representation problem and pre-trains harder. This note proposes an interactive alternative for one concrete setting: a user describes an image they want, in their own language, and a search agent may ask a few yes/no questions before retrieving. Agents that choose questions to maximize expected information gain already work very well in English. Expected information gain is the expected drop in the agent's uncertainty over candidate images once it hears the answer. On GuessWhich, AQM+ cut retrieval error by more than 60% over ten rounds, while supervised and reinforcement-learning questioners cut it by less than 6% (AQM+). These agents fail in a specific way outside English. Their initial belief over images comes from a description the model only partly understands, so the belief is confidently wrong. Bisecting a confidently wrong belief spends questions on the wrong candidates.

The core idea is to make the model's lexical uncertainty explicit and askable. Code-switching here means replacing some words of a sentence with their translations in another language. From the user's description we generate a small set of variants in which different content words are swapped into English. Each variant acts as a competing hypothesis about what the description means to the model. When the variants disagree about which images match, that disagreement pinpoints which words the model fails to ground visually. The agent then tracks a joint belief over pairs of (target image, interpretation) and chooses questions that are informative about both. In two sentences: code-switched paraphrases of a non-English query tell the agent which words it does not understand. An information-gain questioner then asks the user about exactly those visual concepts, instead of trusting a miscalibrated first guess.

Core Claim

In multilingual interactive image retrieval, much of the cross-lingual gap is miscalibrated lexical grounding, and a few targeted questions can recover it. This works only if the questioner's belief represents that grounding uncertainty. We claim three things relative to an English-style information-gain questioner run directly on target-language descriptions, under the same budget of two to five yes/no questions. First, a questioner whose belief is a mixture over code-switched interpretations will reach the target image faster. Second, the advantage will grow as language resources shrink. Third, the advantage will be largest on inputs where machine translation is weakest, namely code-switched and low-resource descriptions.

The mechanism has a simple form. Let $y$ be a candidate gallery image, $m$ one of $M$ code-switched interpretations of the user's description $d$, $q$ a candidate question and $a \in \{\text{yes}, \text{no}\}$ the answer. The agent scores questions by

$$\text{EIG}(q) = H(Y, M \mid d) - \mathbb{E}_{a \sim p(a \mid q, d)}\big[H(Y, M \mid d, q, a)\big],$$

where $H$ is Shannon entropy and $p(y, m \mid d) \propto p(m)\, s(y \mid m)$. Here $s(y \mid m)$ is the retrieval model's score for image $y$ under interpretation $m$, and $p(m)$ is a prior over interpretations. Because the answer depends only on the image, $p(a \mid q, y)$, it can be computed by an answer-prediction model in English on a fixed, translated question inventory. This sidesteps the problem that the agent's internal answerer is weakest in exactly the languages we care about. The file flags that weakness as the main brittleness of information-gain questioners.

Relation to Prior Work

Three goal-oriented visual dialogue systems supply the method. AQM+ approximates information gain over the top $K \approx 20$ candidate images with a simulated answerer. SEEKER keeps a distribution over questioning policies and reaches 74.4% on new GuessWhat?! objects. INVIGORATE asks clarification questions inside a planning loop for robotic grasping, averaging 0.65 questions per episode at 83% success. All three are English-only, and all three put uncertainty only over targets. We add an interpretation dimension whose source is cross-lingual grounding failure.

Mario does multimodal multi-turn clarification for search with a fine-tuned LLaVA reranker, reaching 59.36% mean reciprocal rank on ClariMM. It generates questions by fine-tuned prompting rather than by information gain, and it works in English.

Code-switching work motivates the probe and also constrains how we use it. CoSDA-ML and M²-VLP use random lexical swaps as training augmentation. In M²-VLP this was the largest single gain for multilingual vision-language transfer, but its t-SNE plots show low-resource languages such as Swahili drifting away from English in visual space. That drift is evidence that the grounding gap is word-level and language-dependent.

Attention-informed mixed-language training and saliency-based mixed-language training choose which words to swap using attention or gradients, again at training time. Toward the Limitation of Code-Switching shows that dictionary swaps produce ungrammatical strings and wrong word senses. That warning matters here. We never train on the variants and never present them to the user. We use them only at inference, as inputs to a retrieval scorer. Sense errors in the dictionary then become wrong interpretations, which the questioning can rule out, rather than corruption baked into the weights.

The cross-modal attention and representation-gap literature supplies the scorer and the diagnosis. We use a cross-attention reranker in which text tokens attend to image regions. The reason is that MAF shows region-level features are what disambiguate entities that share a class label (61.43% vs 38.71% on Flickr30k Entities). Learning Robust Visual-Semantic Embeddings argues that instance-level contrastive training leaves global geometry unaligned. We expect this to show up as overconfident wrong beliefs, and we measure it directly.

Research Program

The system has four parts. The first is a multilingual cross-attention retriever over a gallery, producing $s(y \mid m)$. The second is an interpretation generator. For a description in language $\ell$, it uses bilingual dictionaries to produce $M$ variants (we will try $M$ from 4 to 16), each swapping a different subset of content words into English. It also includes the unmodified original and a fully machine-translated variant. The third is a fixed inventory of a few hundred attribute and relation yes/no questions, human-translated once per language, plus an English answer predictor $p(a \mid q, y)$. The fourth is the joint belief update and information-gain selection over the top-$K$ images crossed with the $M$ interpretations.

The cost per turn scales with $K \cdot M$ times the question inventory size. That is larger than AQM+'s cost, which already ran at about 3 seconds per turn against 0.1 seconds for a feedforward questioner (AQM+). We will report wall-clock latency alongside accuracy.

The analysis component asks whether variant disagreement actually tracks grounding failure. For each language, we will compare the entropy over $M$ with the per-description retrieval error. We will also check whether the words most often swapped in high-disagreement interpretations are the ones whose region attention is diffuse. If disagreement does not predict error, the core claim fails at its first step, whatever the downstream numbers say.

Baselines

The naive reference point is single-shot retrieval from the target-language description with the same multilingual retriever and no questions. It shows how much interaction buys at all.

The strongest passive competitor is Translate-Test single-shot retrieval: machine-translate the description to English and retrieve with an English model. IGLUE reports that this often beats multilingual pre-trained encoders. Among those encoders, UC² was the strongest. M²-VLP reports a 10.6% average zero-shot retrieval gain over UC² and M³P on IGLUE. I found no reported absolute xFlickr&CO retrieval figure on Lacuna for either model, so we will reproduce both baselines ourselves on identical galleries.

The strongest interactive competitor is Translate-Test plus an AQM+-style questioner. It translates the description, builds the belief in English, and selects questions by information gain over images only, using the same question inventory, answer predictor, $K$, and turn budget. Its English reference number is AQM+'s 98.25% percentile mean rank at round 10 on GuessWhich, against 95.69% for RL-QA (AQM+). Our setting uses a different gallery and fewer turns, so this number is context, not a target.

A second interactive baseline runs the same image-only questioner on the untranslated description. Three ablations isolate the key ingredient:

  • Model-uncertainty ensemble. Interpretations are replaced by five retriever checkpoints trained with different seeds. This keeps a mixture but removes the word-localized source.
  • Random code-switch variants. The same $M$ is used, with uniformly random swaps.
  • Prior-only mixture. We average over interpretations to form the prior but drop $M$ from the information-gain objective, which tests whether asking about interpretations matters beyond ensembling.

All systems share the retriever, gallery, question inventory, answer predictor and user simulator, and each gets the same hyperparameter search budget on English development data only.

Beating the strongest baseline would mean higher recall@1 after three questions than Translate-Test with AQM+-style questioning. The gap would need to be significant across three seeds, averaged over the lower-resource half of the languages. It would also need to show no meaningful loss on high-resource languages.

Why not just…?

Why not just translate everything to English and run an English questioner? This is the real threat. On high-resource languages with clean descriptions we expect it to tie or win, and we will report that honestly. The case against it rests on inputs where translation is unreliable. The file notes that translation tools struggle with code-switched text, low-resource varieties and domain terms (EntityCS, xGQA). The decisive comparison is therefore stratified by language resource level and by input type. Our system also contains the full translation as one interpretation, so where translation is right the posterior should concentrate on it. If Translate-Test still wins on low-resource and code-switched inputs, the proposal is wrong.

Why not just fine-tune on a few target-language examples? xGQA reports that as few as 48 images with translated questions recover up to 20 accuracy points. That shows cheap supervision is powerful. We will include a few-shot fine-tuned retriever as an additional baseline and also run our questioner on top of it, since the two are complementary. Our claim is narrower: for languages and inputs with no labeled data, interaction substitutes for that supervision at a cost of two or three questions.

Why not just use ensemble or dropout uncertainty instead of code-switched variants? An ensemble would also make the belief less overconfident. What it cannot do is say which word is ungrounded, and that is what makes a question targeted. The ensemble ablation in the Baselines section tests exactly this. If it matches the full system, the code-switching component adds nothing and should be dropped.

Why not just train a question generator by supervision or reinforcement learning? On GuessWhich, such questioners barely improve beyond the initial caption: less than 6% error reduction versus more than 60% for information gain (AQM+). Single-policy reinforcement learning also collapses to repetitive questions (SEEKER). No multilingual training dialogues exist for such a generator anyway.

Proposed Experiments and Decisive Checks

The main testbed turns xFlickr&CO from IGLUE into interactive retrieval. Its target-language captions serve as user descriptions, and galleries of 1,000 images are drawn with hard negatives that share caption keywords. Languages will be grouped by pre-training resource level. We will report:

  • recall@1 and percentile mean rank after 0 to 5 questions;
  • the number of questions needed to reach the target;
  • expected calibration error of the initial belief per language.

The user simulator answers yes/no from a different, stronger vision-language model than the agent's internal answer predictor, to avoid collusion between simulator and agent. A human study with native speakers in two low-resource languages, 100 searches each, checks that simulated gains survive with real answers.

Real code-switched descriptions are the regime where we expect the biggest win. Our method generates synthetic code-switching, so a synthetic code-switched test set would be circular. We will instead collect a few hundred naturally code-switched descriptions from bilingual annotators. As a transfer check, we will apply the same interpretation layer to xGQA, scoring answer candidates rather than images.

Two checks decide the proposal. First, interpretation entropy must predict per-description error, and the full system must beat the ensemble and random-swap ablations. Without both, the mechanism is not what we claim. Second, the full system must beat Translate-Test with AQM+-style questioning on low-resource and natural code-switched inputs at three questions. Without that, the simpler pipeline is sufficient and the proposal should be abandoned in its favor.