Robots that follow spoken directions are trained almost entirely in simulated buildings and almost entirely in English. People usually treat these as two separate problems. The first is that simulated camera images are cleaner than real ones. The second is that non-English instructions are understood less well. This proposal argues the two interact. When an agent understands an instruction poorly, as tends to happen in Hindi or Telugu, it leans harder on what it sees. If what it sees was learned from a simulator that misrepresents real cameras, the visual error costs more. The prediction is that moving from simulated to real images should hurt a navigation agent more for non-English instructions than for English ones. It also predicts that simulator settings tuned on English will rank multilingual agents wrongly.
To test this cheaply and repeatably, we will build a paired benchmark. It uses the CODA office, which already has both a physical space and a scanned "digital twin." At every navigation waypoint in the twin we capture real 360° camera panoramas, and we write instructions for the same paths in English, Hindi, and Telugu. Any agent can then be run on the same instruction and path twice, once with simulated images and once with real ones, without moving a robot. This separates the perception gap from the motion-control gap, measures how the perception gap differs by language, and lets us pick training-time image corruptions by how well they predict real performance rather than by how well they score in simulation.
Core Claim
The claim has two parts.
The first is empirical. In vision-and-language navigation (an agent receives a natural-language route description and must move through a building to the described goal), the drop in success from simulated to real images is larger for Hindi and Telugu than for English. This difference persists after controlling for the content of the instruction.
The second explains why, using the idea of model misspecification: the true process generating the data lies outside what the model can represent. A simulator is a misspecified model of real sensors. A policy trained on it fits the simulator's visual statistics. The tighter that fit, the more the policy is exposed when the statistics change, and the fit is tighter when the language channel carries less usable information.
This reasoning yields a measurable, falsifiable prediction. Across languages, and across instruction-degradation conditions inside a single language, the real-image penalty should grow as the agent's measured reliance on language falls.
A practical consequence follows from Sim2Real Predictivity. That work showed that tuning simulator parameters to maximize the correlation between simulated and real rankings (raising the success-rate Sim2Real Correlation Coefficient (SRCC) from 0.18 to 0.844) works better than adding hand-built noise models, which actually lowered that correlation. If the perception gap depends on language, the best simulated-perception settings also depend on language, and English-calibrated settings will misrank Hindi and Telugu agents.
Positioning Against Prior Work
Three papers motivate the question, and each leaves out exactly the combination we study.
Sim-to-Real Transfer for Vision-and-Language Navigation deployed an English instruction-following agent in the physical CODA space using color-jitter augmentation and a waypoint-predicting network. It reports 55.9% success in simulation, 46.8% on the robot with a map, and 22.5% without one. It attributes most real failures to predicting where the robot can physically go rather than to visual differences. We take that finding seriously. It is why our benchmark replays real images on the twin's fixed waypoint graph: doing so removes the action-space gap and isolates the perceptual one, which that paper could not separate cleanly.
Room-Across-Room supplies the multilingual protocol and data. It collected 126,000 English, Hindi, and Telugu instructions from native speakers and found agents reaching roughly 25 to 30% success against 94% for humans. It also found that a single multilingual model underperformed separate monolingual ones. Its evaluation was entirely in simulation.
CLEAR aligns instructions for the same path across languages and matches detected objects across buildings, improving nDTW by 2.59% over CLIP-based monolingual baselines and unseen-environment success by 5.3%. It is the strongest existing method for cross-lingual navigation. It assumes perfect rendered images and never meets a real sensor.
None of these three measures how the language and sensor gaps interact.
The misspecification literature constrains the design. The Dipping Phenomenon shows that when a model class cannot represent the true distribution, more training data can drive error up toward chance once performance passes an optimum at finite sample size. The comparable risk here is that more clean simulated training makes real-image performance worse, so we track real-image learning curves rather than assuming monotone improvement.
Learning Graphical Model Parameters with Approximate Marginal Inference shows that objectives fit to the quantities you will actually be scored on can "learn around" misspecification. That supports selecting training corruptions by real-image predictive validity rather than by fidelity to an assumed noise model.
Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control is a cautionary result for any uncertainty-based fallback. Monte Carlo dropout's safety score fell to 0.00 under rain in the CARLA driving simulator. We therefore do not rely on dropout uncertainty as a reliance measure.
Research Program
Building the benchmark. The benchmark has three components.
The real-image graph comes first. At every node of the CODA twin's navigation graph we capture 360° panoramas with the same consumer camera class used in the TurtleBot deployment. We capture under three lighting conditions, plus one pass taken while the robot is moving to include motion blur. The pose of each panorama is registered to its node, so the real images can stand in for the rendered ones at each step.
The second component is the instructions. Native speakers write instructions for the same set of CODA paths in English, Hindi, and Telugu, following the Room-Across-Room collection protocol. As a separate control, professionally translated versions of the English instructions are also prepared. Native instructions capture real linguistic and cultural variation in how routes are described. Translated instructions hold content fixed, so any remaining language effect cannot be blamed on Hindi speakers simply describing routes differently.
The third component is a small physical-robot check, using the waypoint pipeline from Anderson et al., with a map.
Measuring the interaction. For each agent and language we compute success rate and nDTW, a measure of how closely the agent's path follows the reference path, on simulated and on real images. The quantity of interest is the interaction: the real-minus-simulated drop in a given language, minus the same drop in English. We report bootstrap confidence intervals over paths, paired by path.
To test the proposed mechanism rather than only the effect, we measure each agent's language reliance. This is the change in action distribution when the instruction is replaced by an instruction for a different path. We then check whether lower reliance predicts a larger real-image penalty. The check runs across languages and also within English, using instructions with landmark words removed. The within-English test matters. If the penalty tracks reliance there too, the effect is about information in the language channel and not about some property of Hindi or Telugu as such.
Calibrating training corruptions. We train agents with a family of image corruptions applied to Matterport3D renders: color jitter at several strengths, blur, exposure shifts, and lens distortion. Each training configuration is then scored by SRCC between simulated and real-image rankings, computed separately per language on a calibration half of the CODA paths.
The method we propose is to choose the corruption setting per language by this real-image SRCC rather than by simulated success. A small real calibration set is used only for model selection, never for gradient updates. This adapts the Kadian et al. approach of tuning the simulator for predictivity rather than realism, moving it from actuation to perception and making it language-specific.
Baselines
Clean, monolingual baseline (the "do nothing" reference). This is the standard Room-Across-Room agent trained without corruption on clean renders, one model per language. It sets the size of the raw gap. The only reported numbers on Lacuna are simulation success of roughly 25 to 30% from Room-Across-Room. No reported result was found on Lacuna for any multilingual agent on real images. Every agent below therefore uses the same backbone, training episodes, and number of hyperparameter trials.
Color-jitter baseline. This is the domain randomization from Sim-to-Real Transfer for Vision-and-Language Navigation, applied with its English-tuned strength to all languages. It is the established perceptual fix, and the paper's 46.8% with-map robot success is the reference point for English. Our method beats it if per-language calibration reduces the Hindi and Telugu real-image drop without increasing the English one.
CLEAR (the strongest competitor). CLEAR is the strongest existing multilingual method. Its alignment across languages could, by itself, raise non-English language reliance and shrink the interaction. We retrain it on our splits with its published two-stage recipe and an equal tuning budget. We run it both alone and combined with our calibrated corruptions. Its reported gain of 2.59% nDTW over CLIP-based monolingual baselines is a simulation number. Beating it means a lower non-English real-image drop at matched simulated performance.
Ablation of the key ingredient. The ablation applies identical corruption families, but selects the setting by simulated success rather than by real-image SRCC. It also tests a single English-calibrated setting against per-language settings. This isolates whether the gain comes from selecting by real predictive validity in a language-specific way, or merely from augmenting at all.
Why not just…?
Why not just translate Hindi and Telugu instructions into English and use the English agent? This is a genuine threat. If translating at test time removes the interaction, the problem reduces to multilingual text understanding, and perception calibration per language is unnecessary. We run translate-at-test-time as a full condition. Its result is diagnostic either way. If the interaction disappears, the mechanism is confirmed but the fix is translation. If the interaction persists, which is plausible because Room-Across-Room found that native speakers describe routes differently, then translation is not sufficient. The native-versus-translated instruction contrast decides between these two outcomes.
Why not just use stronger, uniform domain randomization for everyone? Anderson et al. already found randomization alone insufficient. Kadian et al. found that a misspecified noise model lowered sim-to-real correlation below that of clean simulation. Stronger randomization can therefore hurt. Our corruption sweep plots real-image success against corruption strength for each language. If the curves peak at the same strength in every language, uniform randomization suffices and our per-language claim is refuted.
Why not just fine-tune on the real CODA panoramas? With a few hundred real views from one building, fine-tuning risks memorizing that building. It also removes the held-out real evaluation the benchmark exists to provide. We include it as an upper reference anyway: we fine-tune on calibration-split nodes and test on held-out regions. If it closes the gap in every language, the practical recommendation becomes "collect a little real data," and we will say so.
Decisive Checks
The central test. This is the paired interaction estimate on held-out CODA paths. The hypothesis fails if the real-image drop for Hindi and Telugu is within the confidence interval of the English drop, under both native and translated instructions.
The mechanism test. This is the within-English reliance experiment. If removing landmark words does not increase the real-image penalty, the misspecification explanation is wrong even if the cross-language effect exists.
The predictivity test. We compare per-language SRCC between simulated and real-image rankings, computed across at least fifteen trained configurations, for an English-calibrated setting versus a per-language setting. The predicted signature is that English-calibrated SRCC is high for English but noticeably lower for Telugu.
The dipping test. We train on 10%, 25%, 50%, and 100% of clean Room-Across-Room data and plot real-image success per language. A rise followed by a fall on real images, while simulated success keeps climbing, would be the first evidence of dipping in embodied sim-to-real training. It would also justify early stopping on a real calibration set.
Physical confirmation. We run roughly thirty robot episodes per language for the best and worst configurations. This checks that the real-image replay ordering holds on hardware. Anderson et al. found the action gap dominant, so we expect robot success to be lower than replay success but the ranking between configurations to be preserved.
References
- Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?
- Sim-to-Real Transfer for Vision-and-Language Navigation
- Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
- CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations
- The Dipping Phenomenon
- Learning Graphical Model Parameters with Approximate Marginal Inference
- Uncertainty Quantification with Statistical Guarantees in End-to-End Autonomous Driving Control