Convention-Aware Consensus for Multilingual Spatial Data

Vision-language models, systems that answer questions about images, often cannot tell "left of" from "right of". The problem gets worse outside English. Speakers of different languages use different rules to map a scene onto words like "in front of" or "to the left". An evaluation across 109 languages found that current models apply English-style rules to nearly all of them. The obvious repair is to collect spatial training data in those languages: native speakers judge model-drafted descriptions of rendered 3D scenes, and a filter keeps only the labels most annotators agree on. We argue that this obvious repair fails in a specific, predictable way. When a language community mixes spatial conventions, majority agreement erases the minority convention along with the genuine errors, and it can push the data back toward the same English default the model already has.

Our proposal keeps human verification but changes what "consensus" means. The scenes are synthetic, so their exact geometry is known. For every candidate convention, such as "left" from the camera's view or "left" from the object's own front, the correct word follows deterministically from that geometry. So we can ask whether each annotator is consistent with some convention, not whether they agree with the crowd. An annotator who reliably uses a minority convention is kept, and one who contradicts every convention is dropped. A modest number of such judgments per language identifies which conventions its speakers use and when. Geometry then generates as much correctly labeled training data under those conventions as we need. The two-sentence version is this: in spatial language, much annotator disagreement is structured by convention rather than noise, and because rendered scenes have known geometry, we can separate the two and filter only the noise. That lets humans supply the cultural rule while programs supply the scale.

Core Claim

Spatial terms are resolved relative to a frame of reference, the viewpoint or anchor object from which "left", "front", and "behind" are computed. Two findings motivate the work. Do Vision-Language Models Represent Space and How? (COMFORT) rendered Blender scenes, rotated objects in 10° steps, and prompted models in 109 languages. It found that GPT-4o imposes an English-style "reflected" projection and a camera-centred view even for languages such as Hausa and Tamil, where speakers use translated or rotated projections. It also found that consistency between "left" and "not right" is near chance. FoREST found 20–30% accuracy drops when English questions take an object's perspective instead of the camera's.

Both papers diagnose the problem but offer no training intervention. FoREST is English-only by its own account. The data-synthesis side does not cover this case either. SpaRE generated 3.4M English spatial question-answer pairs with automatic CLIPScore-based checks and explicitly leaves frame-of-reference ambiguity open. SpatialVLM generated 2B English pairs from depth-lifted web images. Neither uses human judgment, and neither is multilingual.

The claim is that for spatial language, the right unit of consensus is agreement with a geometric convention, not agreement with other annotators. Filtering under this definition should yield training data that moves models away from the English default in low-resource languages. Standard majority or unanimity filtering on the same human judgments should not.

The mechanism fits in one equation. Scene $i$ has known geometry $g_i$. Each candidate convention $c$ in a small set $C$ has a deterministic labelling function $f_c(g_i)$; the set covers reflected, translated, and rotated projections crossed with camera, addressee, and object anchors. Annotator $a$ produces label $y_{ai}$. We model that label as

$$ P(y_{ai} \mid z_{ak}, \epsilon_a) = (1-\epsilon_a)\,\mathbb{1}[y_{ai} = f_{z_{ak}}(g_i)] + \epsilon_a \,/\, |Y|, $$

where $z_{ak} \in C$ is the convention annotator $a$ uses for scene type $k$ (for example, fronted objects with an addressee present), $\epsilon_a$ is that annotator's error rate, and $|Y|$ is the number of possible answers. The parameters are fit with expectation-maximisation, in the spirit of worker-bias correction (Snow et al.). The key difference is that the hidden variable is the convention, not a single true label. We filter on $\epsilon_a$, and the per-language distribution of $z_{ak}$ becomes the specification for generating training data.

This setup directly addresses two open problems in the filtering literature. The first is loss of legitimate ambiguity (Spam Filtering Distorts Data Label Distributions). The second is that global worker filters discard workers who are competent on minority categories, which LCWF itself acknowledges.

Research Program

We render COMFORT-style scenes with a new, disjoint set of objects and layouts, covering both non-fronted objects (balls) and fronted objects with an optional human addressee. We extend these with vertical and near/far relations, which COMFORT omits. For each scene, a language model drafts candidate sentences in the target language under every convention in $C$.

Native-speaker reviewers only verify: they accept or reject each sentence for the pictured scene and can flag wording that is ungrammatical. This author-reviewer pattern follows Role Specified Dialogue, which reports review to be about 13 times more efficient than authoring. Two adaptations address known weaknesses of the pattern:

  • Anchoring. Model drafts could steer reviewers toward English conventions. Every convention is drafted equally often and shown in random order, so reviewers are never nudged toward one reading.
  • Fatigue. Geometrically unambiguous control items, such as "on top of", are interleaved to track each reviewer's error rate over time.

We target six languages. English serves as a control. Hausa and Tamil are the languages COMFORT singles out as non-reflected. Swahili and Urdu are low-resource languages named in the multilingual-disparity literature. Chinese adds a high-resource comparison. We do not assume which conventions any of these languages uses beyond what COMFORT reports; that is an output of the study. We collect roughly five judgments per item, which is above the approximately four that Snow et al. found sufficient for expert-level quality.

After fitting the model above, we generate training pairs from geometry under each language's inferred convention mixture, with the scene type conditioning the convention. We then fine-tune Qwen2-VL-2B and 7B, the models SpaRE used, on a shared multilingual mix. A small English SpaRE-style replay component guards against regression in English.

Baselines

The naive reference point is the base Qwen2-VL model prompted directly in each target language. It tells us where the English default sits for this model family.

The strongest existing data method is SpaRE-style synthetic English spatial data. Its reported result is a 49% improvement on What'sUp for Qwen2-VL-2B, plus an average 9% improvement on VSR and 3DSRBench (SpaRE). We run this baseline in two forms. The first uses the English data as released. The second machine-translates it into each target language, matching our example count and fine-tuning budget.

For COMFORT itself, we found no reported scalar result on Lacuna. The paper reports only that opposite consistency is "often no better than random" and that models default to English conventions. We will therefore re-run COMFORT on every condition ourselves. Beating SpaRE means two things together: higher agreement with each language's native-speaker convention on COMFORT and on our held-out native-speaker test set, and English VSR and What'sUp accuracy no worse than the SpaRE-trained model.

The key ablations hold the human judgments, scenes, training-set size, and compute fixed and vary only the filter:

  • no filtering, keeping all labels;
  • majority vote;
  • unanimity, as in Yong et al.;
  • LCWF worker filtering. Its reported gain on the noisy Music_genre dataset was majority-vote accuracy rising from 71.43% to 79.14%.

Each ablation is followed by training on the surviving labels, or on geometry-generated data under the majority convention. The hypothesis is that convention-aware filtering matches the others in English, where one convention should dominate. It should beat them specifically in languages with mixed conventions and on minority-convention test items.

We add two prompting baselines. One is Cross-Lingual-Thought prompting (XLT), which lifted gpt-3.5-turbo on MGSM from 23.3 to 70.0 by reasoning through English. The other is an oracle prompt that states the target convention explicitly.

Why not just…?

Why not just translate existing English spatial data? Translation copies the English frame of reference word for word, so it teaches the model the wrong rule in exactly the languages where the rules differ. COMFORT's finding that models already apply English conventions everywhere suggests this default is what current data produces. The translated-SpaRE baseline tests this directly. If it matches our method on COMFORT convention agreement, the cultural-convention story is wrong for these languages.

Why not just reason through English with XLT? XLT's own ablations show that its gains depend on translating into English. For spatial words, that step is the failure point, because the English rendering of "left" silently fixes the frame. We expect XLT to raise overall fluency and question-answering accuracy while lowering convention agreement for non-reflected languages. The COMFORT comparison measures exactly this.

Why not just tell the model the convention in the prompt? This is the most serious threat. If an oracle convention prompt closes the gap, the training intervention is unnecessary. The data-collection pipeline would still be valuable, but only as a way of discovering conventions. Two things argue against this outcome. COMFORT reports near-chance left/not-right consistency, which suggests the model lacks a stable geometric transform to follow even when instructed. FoREST shows large English-only drops under object-centred frames even with disambiguated prompts. We will report the oracle prompt as a first-class result either way.

Why not skip humans and label from geometry under conventions taken from the literature? Geometry alone is cheap and exact, and our method uses it for scale. Which convention applies, for which scene type, and for which speakers, is an empirical linguistic fact, though. The file documents this only coarsely, through COMFORT's per-language notes. We test the alternative with a condition that assigns each language a single convention from COMFORT's description. If that condition equals ours, the human signal adds nothing beyond confirming the prior.

Proposed Experiments

The first decisive check is whether annotator disagreement actually decomposes into convention plus noise. We measure the held-out likelihood of human labels under the convention mixture model against a single-true-label model such as Dawid-Skene. In English on non-fronted scenes, we expect nearly all annotators to share one convention, which makes English a sanity check. We also count how many annotators LCWF or majority vote would discard who are self-consistent under a non-majority convention. That count directly measures the harm we claim standard filtering causes.

The second check is downstream. We evaluate every trained model on the following:

  • COMFORT convention agreement per language, meaning the fraction of 10° rotations where the model's acceptability judgment matches the native-speaker convention for that scene type;
  • COMFORT opposite consistency;
  • accuracy on a held-out native-speaker test set of about 500 items per language, including a small subset of photographs to test transfer beyond renders;
  • English VSR (humans 95.4%, CLIP 56% per Liu et al.), What'sUp (humans 99%, best model 56% per Kamath et al.), and MMBench, to catch regressions.

The result that would confirm the proposal has three parts. Convention-aware filtering beats majority and unanimity filtering on minority-convention items in mixed languages. It ties them in English. It beats both translated SpaRE and XLT on COMFORT for Hausa and Tamil without lowering English VSR.

The result that would refute the proposal is any of three outcomes. Standard filtering matches it everywhere. The oracle prompt matches it. Or literature-assigned conventions match it. Any one of these would mean the structured-disagreement premise does not buy anything in practice.