Many language-guided agents decide step by step: a navigation robot picks the next waypoint based on the instruction and on where it has already been, and a retriever picks the next passage based on the passages it has already chosen. These history-conditioned models beat models that score each candidate independently. When they are trained in simulation and deployed in the real world, the usual fix targets the camera image: make simulated pixels look real, or use features that ignore appearance. This proposal argues that part of the sim-to-real gap sits somewhere else, in the agent's own record of past decisions. In simulation that record is mostly correct, so the model learns to trust it. In reality, early perception mistakes are more frequent and of different kinds, the record fills with wrong entries, and a model that trusts its history compounds each error.
The proposal has three parts. First, measure how much of the real-world performance drop comes from corrupted history rather than from unfamiliar observations. Second, train in simulation with histories corrupted to match the error pattern observed on a small real calibration set. Third, add a learned gate that down-weights history entries that disagree with the current observation. The sharpest test uses a physical office that has an exact simulated twin. The twin makes it possible to swap histories between the real and simulated runs and read off each channel's contribution directly.
Core Claim
For sequential decision models, the sim-to-real gap has two separable components. The observation gap is that current inputs look different. The history gap is that the distribution of the agent's own upstream errors differs between sim and real. Our hypothesis is that the history gap accounts for a large share of the collapse seen in long-horizon transfer, and that observation-side fixes leave it untouched.
The evidence motivating this comes from Sim-to-Real Transfer for Vision-and-Language Navigation. That work moved an instruction-following navigation agent from Matterport3D simulation to a TurtleBot2 in the "Coda" office and its digital twin. Success rate fell from 55.9% in simulation to 46.8% on the robot with a map, and to 22.5% without one. The authors traced failures to wrong waypoint predictions, such as missed narrow gaps between furniture, rather than to visual differences. Our reading is that an early wrong waypoint poisons every later decision.
Three results on the contextual side explain why history matters but do not test it under shift:
- SeqGROUND grounds phrases to image regions with a stack that remembers earlier matches, and removing that stack drops Flickr30K Entities accuracy from 61.60% to 59.55%.
- Multi-Hop Dense Retrieval conditions each retrieval step on earlier passages and finds that order-agnostic training loses substantial accuracy.
- Suhr and Artzi attend over instruction and state history to resolve references on SCONE.
All three evaluate on in-distribution data, where upstream decisions are mostly right. None asks what happens when the history itself comes from a shifted domain.
Research Program
The diagnostic. The Coda digital twin allows a crossover experiment. We record real robot episodes, including the waypoints the agent chose. We then replay each episode's history into the twin: the agent sees simulated observations but carries the real, error-laden history. Conversely, we run the real robot with a history taken from the clean simulated rollout of the same instruction. Comparing the four cells (sim observations with sim history, sim with real, real with sim, real with real) splits the success-rate drop into an observation term, a history term and an interaction. A large history term is the precondition for everything that follows. If it is small, the proposal reduces to observation-side adaptation, which prior work already covers.
The method. A history error model $q(\tilde h \mid h)$ maps a clean history $h$ to a corrupted one $\tilde h$. We fit it from a small real calibration set of a few dozen episodes by recording, per step, which kinds of mistakes occur and how often: an adjacent waypoint instead of the correct one, a skipped doorway, or a wrong referent. During training in simulation, histories are sampled from $q$ rather than taken clean. A reliability gate $g(\tilde h_i, o_t) \in [0,1]$ scores each past entry $\tilde h_i$ against the current observation $o_t$, and the policy attends to history weighted by $g$. The training objective is
$$ \mathcal{L} = \mathbb{E}_{h \sim \text{sim},\ \tilde h \sim q(\cdot \mid h)} \big[ \ell(\pi(o_t, x, g \odot \tilde h),\ a_t^\ast) \big], $$
where $x$ is the instruction, $\pi$ is the policy, $a_t^\ast$ is the correct action from the true simulator state, and $\ell$ is cross-entropy.
The correct action comes from the simulator, so the policy learns to recover from realistic wrong histories. The gate receives supervision for free, because the simulator knows which corrupted entries are wrong. This is the key difference from on-policy training in simulation: there, the agent recovers only from the errors it makes in simulation, which by the Anderson et al. diagnosis are not the errors it makes in reality.
The design borrows a mechanism from Adversarially Masking Synthetic to Mimic Real. That work learns differentiable masks that drop points from overly complete synthetic LiDAR scans so they resemble real sensor dropouts, raising SynLiDAR→SemKITTI mIoU from 15.2% to 22.8%. We apply the same "make synthetic imperfect in the way reality is imperfect" idea to the decision record instead of the sensor. As an optional variant, we test an adversarial version: masks over history entries are chosen to maximize loss while being constrained to match $q$'s marginal error rate. The method is compatible with the counterfactual step-wise rewards of Suhr and Artzi, which require a simulator. Our corruption happens entirely inside the simulator, so that constraint is respected.
Relation to other sim-to-real work. These methods sit upstream of ours and are complementary:
- FAINT reaches 90% real-world success in image-goal navigation using frozen appearance-invariant features and on-policy correction in simulation, but it has no language instruction or linguistic history.
- AnyCamVLA keeps a manipulation policy frozen and re-renders test cameras into the training viewpoint.
- PixelDA translates synthetic pixels toward real ones.
All three repair the observation channel only. MS-VAE aligns latent codes of synthetic or unpaired trajectories with real paired ones, reaching 56.1% success on unseen R2R. It aligns whole-trajectory representations inside simulation, whereas we model the per-step error process and validate it on hardware.
Proposed Experiments
The primary testbed is the R2R-trained agent transferred to the Coda office and its digital twin, using the protocol of Anderson et al.. We evaluate both the mapped and cold-start settings with success rate as the metric. Calibration episodes are disjoint from test instructions.
The secondary testbed is controlled and cheap to reproduce: LIBERO under unseen camera viewpoints, treated as a stand-in for observation shift. It uses a history-conditioned manipulation policy with multi-step language tasks. Here the "real" calibration set is a small number of held-out-viewpoint episodes.
A third, text-only sanity check adds realistic retrieval errors to earlier hops of Multi-Hop Dense Retrieval on HotpotQA, measuring supporting-passage exact match. This tests whether calibrated history corruption helps a non-embodied sequential model when upstream steps are noisy.
Beyond task success, we report a history-reliance score. It is the accuracy drop when true history is replaced by $q$-corrupted history, measured at each step index. This shows whether errors compound with horizon length.
Baselines
Naive reference: the unmodified sim-trained agent. This is the Anderson et al. pipeline run as published: waypoint predictor with color-jitter randomization plus a classical controller. Its reported success rates of 46.8% with a map and 22.5% cold start are the floor. We retrain it on our compute budget, with the same R2R data and hyperparameter search budget as every other arm, and confirm we reproduce these numbers before comparing.
Strongest existing competitor for the history problem: on-policy correction in simulation with frozen appearance-invariant features. This follows FAINT and is adapted to instruction following by adding the instruction and history inputs. It is the strongest competitor because it already trains on imperfect histories and already shrinks the visual gap. The file reports its 90% real-world success only for image-goal navigation, and no reported result on Lacuna was found for this recipe on Coda. We run it with the same sim data, the same number of training steps, and the same tuning trials.
Strongest observation-side competitor on LIBERO: AnyCamVLA. It reports 94.5% average success on unseen viewpoints versus 67.9% for the base policy. We run it both alone and combined with our method. On LIBERO, beating it means either exceeding 94.5% when combined, or showing that the remaining failures are history-driven by the reliance score. Since this is a controlled viewpoint benchmark rather than real transfer, we expect the margin there to be small.
Parallel reference: a history-free agent. This agent conditions only on instruction and current observation. It tests whether history is simply a liability under shift, a possibility the SeqGROUND ablation leaves open because that ablation was measured only in-domain.
Ablations isolating the key ingredient. Calibrated corruption without the gate, the gate without corruption, and uniform corruption at the same overall error rate. The uniform-corruption ablation is what tests the claim that the shape of the real error profile matters, not just its rate.
What counts as success. On Coda cold start, a success-rate gain over the on-policy baseline that exceeds the variation between random seeds, with the gain concentrated in episodes whose early steps contained errors.
Why not just…?
Why not just run on-policy correction (DAgger) in simulation? On-policy correction teaches recovery from the agent's own mistakes, but the mistakes it sees are simulator mistakes. Anderson et al. report that the real failures are of a type the simulator's navigation graph under-produces, namely missed tight passages. Whether sim-generated errors cover real ones is empirical, and this alternative is a real threat. The crossover diagnostic settles it: if on-policy training already shrinks the history term to near zero, our corruption model adds nothing and we will say so.
Why not just apply random history dropout or scheduled sampling? This is cheaper and needs no real calibration data. The uniform-corruption ablation at a matched error rate is exactly this alternative. If it matches calibrated corruption, the calibration step is unnecessary, and the contribution shrinks to the diagnostic plus the gate.
Why not just fix the observations, so the history stays clean? Better perception reduces upstream errors, and FAINT and AnyCamVLA show large gains from that route. However, Anderson et al. found the dominant errors were in action prediction, not appearance, so perfect-looking inputs would still leave wrong waypoints in the record. We test this by stacking our method on top of the strongest observation fix and checking whether the gains add up.
Why not just drop history? In-domain evidence says history helps: the SeqGROUND ablation, MDR's ordering result, and the reference resolution reported on SCONE. Under shift it might not. The history-free baseline measures this directly. A win for it would itself be a publishable negative result about history-conditioned agents in transfer.
Decisive Checks
The proposal stands or falls on three outcomes:
- The crossover on the Coda twin must show a history term comparable in size to the observation term.
- Calibrated corruption must beat uniform corruption at the same error rate.
- Gains on cold-start Coda must concentrate in episodes with early errors.
If the first check fails, the history gap is not real at this scale and we stop. If the second fails, the method simplifies to dropout plus gating.
References
- Sim-to-Real Transfer for Vision-and-Language Navigation
- Neural Sequential Phrase Grounding (SeqGROUND)
- Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
- Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observation
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Synthetic vs. Real Training Data for Visual Navigation
- AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Multimodal Sequential Generative Models for Semi-Supervised Language Instruction Following