Distilling a Clean-State World Model into a Filter That Knows When Its Sensors Lie

Robots trained in simulation often fail on real hardware, and one reason is easy to overlook. Real sensors misbehave in ways that unfold over time. A camera drops frames, a depth sensor returns holes when light hits a shiny surface, a view drifts after the camera gets bumped, and motion blur arrives in bursts. Most sim-to-real recipes treat corruption as noise on each frame independently, so the policy has no principled way to decide that the current frame is bad and it should rely on what it expected to see. We propose training, in simulation, a world model on the robot's exact state. This is cheap because clean state is low-dimensional. We then freeze that model's prediction of how the world evolves and use it as the "expected" half of a learned filter. A camera-based student network learns only two things. It learns how to read corrupted images into the teacher's compact state description, and how much to trust each reading.

The practical claim is that this split gives robustness to temporally structured sensor failures that augmentation cannot provide, and it does so at the low sample cost of privileged-state training. When the camera is unreliable, the frozen teacher dynamics carry the estimate forward. When the camera is reliable, the image corrects it. The student's reported uncertainty should rise and fall with actual corruption. That calibration signal is itself useful for deployment.

Core Claim

Existing privileged-distillation methods make the student relearn dynamics from visually randomized data. Our claim is that a student which reuses the teacher's clean-state dynamics as a frozen prior, and learns only an observation update gated by reliability, transfers zero-shot better under realistic, non-Gaussian, time-correlated sensor corruption. It also needs fewer simulated image samples.

The closest work is TWIST. TWIST trains a DreamerV2 teacher on ground-truth states and distills it into a vision student on domain-randomized images. It uses latent distillation (map pixels to the teacher's latent) and imagination distillation (match imagined rollouts), and it reaches 80% zero-shot success on a real Franka block push versus 30% for Dreamer with domain randomization. TWIST supplies our method. We differ in three respects. The student's dynamics are not learned but taken from the teacher. Training corruption is structured and temporal rather than only lighting and texture. Evaluation targets held-out sensor failure types, which TWIST did not test.

Research Program

The teacher is a recurrent world model trained with online RL on simulator state (object poses, joint angles and velocities). Its policy and value function act on its latent $z_t$.

The student maintains its own latent estimate. Each step, it first propagates the estimate through the frozen teacher transition $p^T(z_t \mid z_{t-1}, a_{t-1})$, where $a$ is the action. It then fuses in an embedding of the corrupted observation $\tilde{o}_t$. The fusion weight comes from a predicted observation precision, much as a Kalman gain does. The student is trained on simulator rollouts by minimizing

$$ \mathcal{L}(\phi) = \sum_t \mathrm{KL}\big(p^T(z_t \mid s_{\le t}) \,\|\, q_\phi(z_t \mid z_{t-1}, a_{t-1}, \tilde{o}_t)\big), $$

where $p^T(z_t \mid s_{\le t})$ is the teacher's belief given clean states $s$, and $q_\phi$ is the student's belief given corrupted observations. The student has trainable parameters $\phi$ only in the observation encoder and the precision head. At deployment the teacher's policy runs on the student's latent unchanged.

This objective has no pixel decoder, so it avoids the capacity waste on backgrounds that Contrastive Recurrent State-Space Models documented for reconstruction models. That paper provides motivation for the design. Our target is the teacher's state latent, which contains the goal position. We therefore also avoid CoRe's own failure mode, which is that contrastive objectives erase static goals; that finding constrains our design. We also sidestep the embedding explosion and collapse that Towards Robust Bisimulation Metric Learning diagnoses in bisimulation learning, because we never learn a metric from rewards.

The corruption generator runs on top of standard visual domain randomization. It injects Markov-chain frame dropouts, depth holes tied to surface reflectance, slow camera-extrinsic drift, burst motion blur, and rolling-shutter shear. For evaluation we hold out entire corruption families, not just parameter ranges.

The main risk is the prior itself. If the frozen teacher dynamics are wrong under randomized physics, the filter propagates wrong predictions exactly when the camera is unreliable. To handle this, we train the teacher under dynamics randomization, following Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. We then test explicitly whether a small learned residual on the prior helps or hurts.

Baselines

Naive reference. The naive reference is an end-to-end pixel agent trained with visual domain randomization plus our corruption generator. We use Dreamer with domain randomization, the setup that reached 30% on block push in TWIST. Its corrupted-sensor score shows how much the problem hurts without structure.

Strongest existing method. This is TWIST, retrained with our identical corruption generator added to its domain randomization. Its reported 80% real block-push success is the number to beat. We re-run it rather than quote it, using the same teacher, simulator steps, network size and hyperparameter-search budget. Beating it means matching its clean-condition success and exceeding it by a clear margin under held-out corruption families on hardware.

Asymmetric actor-critic. The third baseline is Asymmetric Actor Critic, in which the critic sees privileged state and a convolutional actor sees pixels. It reported 5/5 real Fetch picks under visual domain randomization. We give its actor a frame stack and the same corruptions, so it has a fair chance to learn temporal robustness implicitly.

Ablations. The key ablation holds everything fixed except the prior. We compare the frozen teacher prior against a student that learns its own transition. A second ablation removes the precision gate, fixing trust in each observation. If neither ablation loses much, the central claim fails.

Why not just…?

Why not just add noise augmentation to TWIST? This is the real threat. TWIST's student already has a recurrent model and could in principle learn to ignore bad frames. Its dynamics, however, are fit from corrupted image sequences. We expect them to be less accurate precisely during long dropouts or drift, which is when a correct prior matters most. The matched-corruption TWIST baseline and the frozen-versus-learned-prior ablation decide this directly. If augmented TWIST ties us, we will report that the contribution reduces to the corruption curriculum.

Why not just use image augmentation such as random crops and random convolutions? The Distracting Control Suite found that augmentation is "essential for any progress" but insufficient: augmented agents lost nearly half their score at low distraction. Policy Similarity Embeddings showed RandConv alone solving 33.7% versus 52.4% with a learned representation. Per-frame augmentation also has no mechanism for a frame that is missing entirely.

Why not just regress the state from pixels and run a classical Kalman filter under the teacher policy? For rigid blocks this could work well, and we include it as an additional baseline with a hand-tuned filter. Our approach generalizes it. The student regresses onto the teacher's latent, which is available even where hand-specified state and noise models are awkward, such as contact-rich pushing. The precision is learned rather than tuned. If the classical pipeline matches us on block push, we will say so and turn to tasks where hand modeling is harder.

Proposed Experiments

Simulation. We use the Distracting Control Suite dynamic-medium setting plus our corruption generator. CoRe reports reaching peak performance within 500k steps there; no exact return value was found on Lacuna. We report episode return versus simulator steps, and separately count image-based steps, to test the sample-efficiency claim.

Real hardware. We use TWIST's Franka block push and block lift. Held-out physical corruptions are induced on the real robot: a strobing light, a reflective cube, a deliberate mid-episode camera nudge, and software frame dropping on the live stream. The metric is success over at least 20 trials per condition.

Decisive checks. Three results decide the claim. First, we must beat retrained TWIST on held-out corruption families while staying within noise of it on clean runs. Second, the frozen-prior ablation must lose most of that gain. Third, the student's predicted precision must correlate with the actual per-frame corruption level on real data. That correlation measures calibration and is evidence that the filter learned when to distrust its sensors rather than getting lucky.