A robot trained in simulation keeps asking one question once it reaches the real world: should it believe the camera frame it just received, or what it already knows about the scene? In simulation that question barely comes up. Frames arrive on time, nothing glares, and the physics behaves the way the model learned it. On hardware, frames are late or dropped, lighting fools the encoder, and contacts slip in ways the simulator never produced. Today's sim-to-real world models (learned simulators of the scene that a policy plans or acts through) answer this question implicitly. They fold every new observation into a single recurrent state vector. The balance between "keep" and "overwrite" is therefore buried in weights tuned for simulated sensors.
We propose making that balance an explicit, differentiable, per-component decision inside the world model's state, then adapting only that decision on the real robot. The model's latent state is a set of tokens. At every step, each token either copies its value from a memory bank of past internal states or is rewritten from the current observation, through a learned copy gate in the style of the Copy-LSTM used for caption editing. On deployment we freeze everything except a handful of gate biases. We fit those biases to a few minutes of unlabeled real rollouts by minimizing the world model's own prediction error. The bet is that a large share of the sim-to-real gap is a miscalibrated trust between memory and observation. If so, it can be repaired with tens of parameters rather than a new encoder or a full fine-tune.
Core Claim
Our hypothesis is that two parts of the sim-to-real gap mostly look, from inside a world model, like a wrong prior on how reliable observations are and how persistent state is. The first part is sensor and timing faults. The second is slowly varying dynamics offsets. A gate that chooses per token between copying remembered state and writing new state therefore gives a small, interpretable surface for adaptation. On that surface, real-world calibration should beat both encoder-only adaptation and plain attention memory for these two gap types. For the third part of the gap, pure visual appearance, the gate should not help. We state that prediction in advance because a crossover between gap types is the cleanest evidence that the mechanism does what we claim.
How This Differs From Existing Work
Several papers supply parts of the method but leave this question open.
Recurrent world model with tokenized latent states (TokenWM) replaces the single latent vector of a recurrent state-space model with a set of tokens and reads a queue of past states through attention. It reports large gains on LIBERO. However, every token is recomputed at each step, and the work is evaluated only in simulation. Our model reuses its tokenized state and memory bank but adds a gate that can preserve a token as-is. We ask whether that preservation path matters under domain shift.
Show, Edit and Tell supplies the copy mechanism: a continuous switch between copying an encoder's internal states and generating new ones, made differentiable by reparameterized binary masks. We move it from caption editing into a robot world model.
Eidetic 3D LSTM and Input-Cell Attention both show that letting a recurrent cell look back at stored states inside its update counters forgetting and recency bias. This motivates placing the copy decision inside the state update rather than in a layer on top of it.
On the sim-to-real side, three lines of work bracket the proposal. TWIST distills a privileged world model into a vision-based student, but the student is a single-vector model that is static at deployment. simDP freezes a sim-trained action decoder and adapts only the visual encoder with real demonstrations. Its authors note that this cannot correct dynamics or timing mismatches. Found-adapt adapts a small latent prompt in seconds, which is the closest precedent for a tiny adaptation surface. Its prompt is stateless and was only verified sim-to-sim. Our adaptation surface is similarly small but lives in the recurrent memory dynamics, and it is fit without action labels.
Two results constrain the design. Generalizing from Synthetic Data to the Real World documents that networks latch onto simulator quirks. An open memory lookup is a natural place for that to happen, because a simulator with perfectly regular timing rewards copying on a fixed schedule. The research file also flags that soft gates can collapse into always-copy or never-copy shortcuts. Our training recipe is built around both problems.
Research Program
The cell. The world model keeps $N$ latent tokens and a fixed-length memory bank of past token sets, following TokenWM. At step $t$, token $i$ forms two candidates:
- a copied value $\tilde{s}^i_t$, read by attention from the memory bank, which includes its own previous value;
- a written value $u^i_t$, produced from the current observation embedding and the action.
The new state is
$$s^i_t = g^i_t\, \tilde{s}^i_t + (1-g^i_t)\, u^i_t, \qquad g^i_t = \sigma\!\left(a^\top [\tilde{s}^i_t, u^i_t, e_t] + b_{k(i)}\right).$$
Here $e_t$ is the observation embedding, $a$ is a learned weight vector, $\sigma$ is the logistic function, and $b_{k(i)}$ is a bias shared by the group $k(i)$ that token $i$ belongs to. We use on the order of eight token groups, so the biases total tens of scalars. We compare two gate variants. One is the soft gate above. The other is a hard binary gate made differentiable with the reparameterization trick from Show, Edit and Tell. The hard variant makes "copied" versus "written" literally inspectable.
Training in simulation. Training uses the standard world-model objectives: observation reconstruction through a decoder, latent prediction, and reward or progress prediction. A policy acts on the latent state, as in World Model-based Perception for Visual Legged Locomotion.
The key ingredient is gap-structured corruption with randomized rates. Frames are dropped or delayed with random jitter, glare and occlusion patches are injected, and friction and mass are perturbed. The corruption rates are drawn from wide distributions per episode, so that neither always-copy nor never-copy is optimal. This is our defense against gate collapse and against the model learning a fixed simulator schedule. We also add a mild entropy penalty on gate usage averaged over each batch, and we log gate statistics throughout training as a collapse monitor.
Adaptation on the robot. We collect a few minutes of unlabeled real rollouts, either the frozen policy's own attempts or teleoperation. With all weights frozen, we fit only the group biases $b$:
$$b^\star = \arg\min_b \sum_t \mathcal{L}_{\text{pred}}\big(\hat{o}_{t+1}(b),\, o_{t+1}\big) + \lambda \lVert b - b_{\text{sim}} \rVert^2.$$
$\mathcal{L}_{\text{pred}}$ is the world model's next-observation prediction loss on real data, $\hat{o}_{t+1}$ is the predicted next observation, $b_{\text{sim}}$ are the biases learned in simulation, and $\lambda$ keeps the fit close to the simulation prior. Fitting tens of parameters takes seconds. At inference the gate is a single forward pass with no inner-loop gradient steps. This matters for control loops running at tens of hertz, where test-time gradient memories like Titans add latency.
Analysis. The main analysis is a gap decomposition. We measure how much of the transfer loss each adaptation surface recovers, broken down by gap type. The adaptation surfaces compared are gate biases, the encoder, both, and everything.
Baselines
The naive reference is zero-shot transfer of the simulation-trained model with no adaptation. simDP reports 0% success for zero-shot transfer of its sim-trained policies to hardware. We expect a similar floor and will report our own.
The first strong competitor is TokenWM trained with exactly our corruption curriculum but without the copy gate. Its memory attention could in principle learn to discount bad frames on its own. This is the ablation that isolates our key ingredient. It uses the same token count, the same memory length, parameter-matched width, the same simulation data, and the same tuning budget. As a sanity reference in clean simulation, TokenWM reports success rates on LIBERO of 68.2% on libero-spatial and 22.6% on libero-long, against 26.4% and 4.4% for RSSM-XL. Our reimplementation should reproduce TokenWM within a few points on clean LIBERO before any transfer comparison is trusted. Beating this baseline means a statistically clear margin under the hardware and injected-gap conditions while staying within noise in clean simulation.
The second is simDP-style encoder adaptation applied to our own world model. We freeze everything except the observation encoder and fit it on the same real data. simDP fits on demonstrations, so we also give this baseline a matched number of action-labeled demonstrations. simDP reports an average normalized task completion of 0.82 on Stack Three, Three Piece Assembly and Threading, against 0.62 for a standard diffusion policy. It also reports 0.35 on Three Piece Assembly with 10 real demonstrations, against 0.30 for standard DP with 50. Our gate-only adaptation does not need to beat this on visual gaps. It should beat it on timing and dynamics gaps, and combined gate-plus-encoder adaptation should beat encoder-only overall.
The third is TWIST: a privileged teacher distilled into a single-vector student with domain randomization. On real Franka Block Push, TWIST reports 80% success against 30% for Dreamer with domain randomization. We rerun TWIST on our hardware with our randomization ranges. Beating it means exceeding its success on Block Push and Block Lift under the same number of trials, and widening the margin when we deliberately add timing faults.
The fourth is full fine-tuning of all weights on the same real data. It shows how much the small surface gives up, or whether full fine-tuning overfits a few minutes of data.
We also include two ablations. A gate with frozen simulation biases and no real adaptation separates the training recipe from the deployment calibration. A scalar gate on a single-vector recurrent model tests whether tokenization is needed at all.
Why not just…?
Why not just train TokenWM with the same corruptions and let attention handle it? This is the most serious threat, and we treat it as a possible outcome rather than a straw man. Attention reads from memory but always produces a fresh state. Prior evidence that in-cell mechanisms beat external ones comes from other domains. IARNN reports a WikiQA mean average precision of 0.7341 against 0.6881 for outer attention Inner Attention based RNNs, and E3D-LSTM reports a Moving MNIST MSE of 41.3 against 46.3 for PredRNN++. None of that transfers automatically to robots. The gate-ablation experiment settles it. If gateless TokenWM with corruptions matches our model on hardware and adapts equally well by fitting a comparable number of parameters (for example its attention temperatures), the copy gate is unnecessary, and we would report it that way.
Why not just adapt the encoder, as simDP does? For appearance gaps, that is probably the right answer, and we predict it wins there. The simDP authors themselves limit it to visual differences. Found-adapt argues that feature alignment alone fails on dynamics shifts. The injected-gap experiments test this directly. Encoder adaptation should recover appearance gaps but not latency or friction gaps, and gate adaptation should show the reverse pattern.
Why not just use domain randomization with distillation, as TWIST does? Randomization teaches a model to be robust on average. It cannot calibrate to the specific timing and contact profile of one robot. TWIST still needed explicit distillation to move from 30% to 80% success. We include TWIST at full strength. If adding timing faults does not degrade TWIST relative to our model, this argument fails.
Why not just use a test-time gradient memory like Titans for dynamics? Titans adapts its memory weights by gradient steps during the forward pass, which suits drifting dynamics. It costs an inner-loop gradient computation per step and has been evaluated on text and time series, not control. We run a Titans-style memory in the simulated stage, where latency can be measured but does not break the robot. If it clearly beats gate calibration on dynamics gaps, we would recommend a hybrid design: Titans-style updates for dynamics, the copy gate for observation trust.
Proposed Experiments
Stage one: controlled sim-to-sim gaps. We train on LIBERO tasks and evaluate the same tasks under held-out gap profiles of each type separately. Timing gaps are latency jitter and frame dropping at rates outside the training range. Dynamics gaps are friction and mass shifts in graded levels, modeled on Found-adapt's friction and gravity grid; no reported result was found on Lacuna for Found-adapt's per-level numbers. Appearance gaps are unseen textures and lighting. Every baseline and adaptation surface is run on each profile with a matched adaptation-data budget. The metric is success rate recovered as a fraction of the clean-to-shifted drop. The decisive check is the predicted crossover: gate adaptation should recover more on timing and dynamics, encoder adaptation more on appearance, and both together more than either alone.
Stage two: cross-simulator transfer. We use REALM, whose calibrated IsaacSim setup tracks physical Franka results with a Pearson correlation of 0.921 between simulated and real task progression REALM. We train in Robosuite and transfer into REALM's base suite and its visual and behavioral perturbations. This gives a hardware-faithful but repeatable target before robot time is spent. REALM reports correlation rather than per-policy success for our task family, so no reported result was found on Lacuna for a directly comparable success baseline. All numbers in this stage are run by us.
Stage three: hardware. We use a Franka Panda on TWIST's Block Push and Block Lift tasks and simDP's three MimicGen tasks. We run 20 or more trials per condition. Conditions are the robot's native timing and a deliberately degraded camera link with added delay and packet drop. Metrics are success rate for the TWIST tasks and normalized task completion for the MimicGen tasks. Each method also gets an adaptation budget measured in real minutes.
The mechanism check. On hardware we log frame timestamps and dropped-packet events independently of the model. If the gate does what we claim, the fraction of copying tokens should rise measurably on frames that arrived late or were lost, and fall back on clean frames. If gate usage on hardware is unrelated to logged sensor faults while task success still improves, the gain comes from something other than calibrated trust, and the paper's central claim would be wrong in its explanation even if the method works.
References
- Recurrent world model with tokenized latent states
- Show, Edit and Tell: A Framework for Editing Image Captions
- Eidetic 3D LSTM: A Model for Video Prediction and Beyond
- Input-Cell Attention Reduces Vanishing Saliency of Recurrent Neural Networks
- TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer
- Encoder-Adapted Sim-to-Real Transfer of Simulation-Trained Diffusion Policies for Robot Manipulation
- Latent Adaptation of Foundation Policies for Sim-to-Real Transfer
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- World Model-Based Perception for Visual Legged Locomotion
- Titans: Learning to Memorize at Test Time
- Inner Attention based Recurrent Neural Networks for Answer Selection
- REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation