Robots trained in physics simulators often fail on real hardware. This failure is called the sim-to-real gap. One reason gets little attention: the policies watch camera video densely, frame after frame, and the way consecutive simulated frames differ from each other is unlike anything a real camera produces. Rendered frames arrive at exactly the right instant. They have no motion blur, no rolling-shutter skew and no variable delay. A policy fed a stack of consecutive frames can learn to read motion from these clean pixel differences, and that shortcut breaks the moment the policy sees real video. We propose to remove the shortcut instead of randomizing around it. The policy sees the camera only as sparse, timestamped snapshots, a few per second, at irregular times. It gets all fast motion information from proprioception (the robot's own joint readings) and touch sensors, which run at a high rate and whose simulated versions are easier to make realistic.
The claim combines two lines of work that have not yet been joined. Video understanding has shown repeatedly that dropping most frames costs little and can even help. Temporal Segment Networks and ClipBERT both train on a few randomly chosen snippets and outperform dense pipelines. Sim-to-real robotics, meanwhile, still trains visual "student" policies on dense consecutive frames and relies on heavy image randomization to survive transfer. The proposal tests whether sparse, irregular visual sampling is itself a sim-to-real regularizer. It also tests whether timing the snapshots to contact events (the moments a finger touches or loses an object) adds anything beyond simple low-rate sampling. A side benefit is that the visual encoder runs far less often on the robot's onboard computer.
Core Claim
A visuomotor policy distilled in simulation transfers better, at a fraction of the visual compute, if it satisfies three conditions. First, it never receives two camera frames close enough in time to compute visual motion. Second, it is told how old its latest snapshot is. Third, it is trained with snapshot times drawn at random within coarse time segments. The prediction is specific. The gap between simulated and real success should shrink relative to a dense frame-stack policy trained with the same domain randomization, and the shrinkage should be largest on tasks where the scene moves quickly or the camera is occluded during contact.
A secondary claim is that triggering a fresh snapshot when the binary contact state changes beats a uniform schedule at the same average frame rate. This secondary claim is a real risk. If uniform low-rate sampling does as well, we drop event triggering. The contribution then becomes the diagnosis together with the simpler multi-rate recipe.
Relation to Prior Work
The proposal borrows its method from the video side.
- Temporal Segment Networks splits a video into equal segments and draws one random snippet per segment, which keeps the network from fixating on exact timestamps.
- ClipBERT trains end-to-end on a few short sampled clips and beats dense-feature models on MSRVTT retrieval (Recall@1 of 22.0 versus 16.8 for dense HERO).
- Adaptive Keyframe Sampling and VideoEspresso show that keeping only a small fraction of frames can improve reasoning.
None of these papers involves control or transfer across domains. We reuse their per-segment random sampling as a training procedure for a closed-loop policy.
The closest robotics work is KeyWorld. It picks keyframes at kinematic turning points of the arm's trajectory, generates only those frames with a video diffusion model and fills the rest with an interpolator. That gives 5.68× faster generation and 90% versus 38% object-level accuracy on LIBERO-goal. KeyWorld differs from our proposal in three ways. It is a world model rather than a controller. It is evaluated only in simulation. Its keyframe rule ignores contact.
Real-Time Chunking handles inference delay by freezing the actions that will run during computation and inpainting the next chunk of actions. It sparsifies when the policy plans, not what the policy sees, so it is complementary to our proposal. We use it unchanged as the execution layer.
The sim-to-real papers play two roles. Some motivate the gap:
- Learning Dexterous In-Hand Manipulation reports that its vision-based block policy falls from a median of 33 consecutive rotations in simulation to 11.5 on the real hand.
- DECO shows that vision alone suffers under hand occlusion. Adding tactile input raises average success from 72.25% to 82.50%. Even so, every modality runs at the same rate.
Others supply method:
- DoorMan provides the pipeline: a privileged teacher in simulation, then a camera-based student trained by DAgger (an imitation method in which the teacher corrects the student's own rollouts). Its deployed student, however, processes every frame.
- Contact-Aware Neural Dynamics supplies the binary contact-event signal that we reuse as a trigger. That paper evaluates contact at dense uniform steps.
One paper constrains the design. Belief-Based Offline RL shows that naively stacking delayed histories collapses under delay: state-augmentation baselines scored about 0.2 on Adroit Hammer, while its belief model scored about 105. Following that lesson, our policy keeps a learned memory of the last snapshot together with its age, rather than a raw stack.
Research Program
We build a multi-rate student policy with three input streams.
- Fast stream. Proprioception and, where hardware allows, tactile readings at the control rate of roughly 100 Hz. Ordinary dynamics randomization is applied to this stream.
- Slow visual stream. Snapshots are encoded only when taken. Each snapshot token carries an embedding of its age in milliseconds and is held in memory until the next one arrives.
- Action head. A flow-matching action-chunk head, executed with Real-Time Chunking.
During distillation from a privileged teacher, we split each rollout into segments. Within each segment, one snapshot time is drawn at random, in the manner of Temporal Segment Networks. The segment length is itself randomized so that the average rate varies between about 1 and 10 Hz. As a result, the policy never sees a fixed spacing between frames.
At deployment we compare three snapshot schedules at matched average rate: uniform, random, and contact-triggered. Under the contact-triggered schedule, a new snapshot is taken whenever the predicted or sensed binary contact state flips, with a minimum rate as a floor.
The analysis side tests the proposed mechanism directly. We train a linear probe to decode object velocity from the visual features of the dense baseline and of our policy. We also measure how much each policy's actions change when only the temporal rendering properties of its input are perturbed: motion blur, rolling shutter, frame-timing jitter of 0 to 100 ms, and dropped frames, with all per-frame appearance held fixed. If the mechanism is right, the dense policy will be far more sensitive to these perturbations than our policy, and that sensitivity will predict its real-world drop.
Baselines
The naive reference point is the dense frame-stack student. It follows DoorMan's teacher-student recipe and is trained on consecutive frames at the full camera rate under the same visual and dynamics randomization. It shows the size of the gap we are trying to close.
The strongest existing method is the domain-randomization pipeline of Robust Visual Sim-to-Real Transfer for Robotic Manipulation. That paper tunes randomization on a cube-localization proxy task and reports 93% average real-world success across seven manipulation tasks, including stacking, assembling, pushing and sweeping. We reimplement its randomization scheme as the default for every arm of our study, so the comparison isolates the temporal input design. Because 93% leaves little headroom, we report its tasks to check that our policy does no harm. We add contact-rich and dynamic tasks where we expect dense policies to degrade.
All students share the same teacher, the same simulator data, the same number of DAgger iterations and the same encoder architecture. Each gets the same hyperparameter search budget, roughly twenty configurations per arm. Beating the strongest baseline means two things on the hard tasks: higher real success averaged over at least three training seeds, and a smaller ratio of real to simulated success shortfall, both at a visual-encoder rate at least five times lower.
Four ablations isolate the key ingredient:
- Uniform decimation. Low-rate uniform snapshots without age embedding or random timing.
- No age embedding. Our policy without the snapshot-age embedding.
- Timing randomization on a dense stack. Random frame delay and frame dropping applied during training to the dense stack, which still allows close frame pairs.
- Schedule swap. Contact-triggered versus uniform scheduling at matched rate.
Why not just…?
Why not just downsample the camera uniformly? This is the most serious competitor, because a fixed 5 Hz camera with a fast proprioceptive loop already removes most usable frame differences. The video literature suggests that fixed spacing lets models latch onto timing regularities. That is why Temporal Segment Networks randomizes snippet position. KeyWorld also reports that undifferentiated temporal processing dilutes critical moments such as grasping. Still, neither result concerns sim-to-real control. The uniform-decimation ablation settles the question directly. If it matches our full method within seed variance, we report that simpler recipe as the finding and drop the irregular and contact-triggered machinery.
Why not just randomize frame timing while keeping dense history? Randomized delay and frame dropping are a natural extension of domain randomization and cost nothing to add. Even with these randomizations, though, the dense stack still contains many closely spaced frame pairs from which the policy can compute clean simulated motion. Our hypothesis is that the shortcut survives this kind of randomization. The timing-randomization ablation, combined with the velocity probe, tests this. If timing-randomized dense students lose their decodable visual velocity and transfer as well as our policy, then our claim reduces to "randomize timing," and we will say so.
Why not just rely on action chunking to save compute? Chunking reduces how often the policy plans, but not how many visual tokens it reads each time. As Real-Time Chunking shows, synchronous chunk execution also produces stop-and-go motion, about 20% slower than its asynchronous method. We therefore treat chunking as the shared execution layer in all arms rather than as a rival. We report encoder calls per second and end-to-end latency, so that compute savings are attributed to sparse observation alone.
Proposed Experiments
Stage 1: sim-to-sim proxy. We train on LIBERO (130 language-conditioned Robosuite tasks; we use LIBERO-Spatial, LIBERO-Goal and LIBERO-Long) and on ManiSkill2 contact tasks such as peg insertion. We then test in a shifted copy of each simulator. The shifted copy adds rendered motion blur, rolling shutter, frame jitter and stochastic control delay of 4 to 16 steps (the delay range used by the belief-based delay-robust offline RL work), plus friction reduced to 30%, following the State Revisit and Re-explore setup. Stage 1 is cheap and gives a first answer on every ablation. We found no reported result on Lacuna for dense frame-stack visuomotor policies under this kind of temporal rendering shift, so all numbers in this stage are ours.
Stage 2: real hardware. On a real arm with a multi-finger hand, matching the XArm7 and XHand setup of Contact-Aware Neural Dynamics, we run three real-world tasks from Robust Visual Sim-to-Real (stacking, assembling, sweeping) as a no-harm check against the reported 93% average. We add three harder tasks: peg insertion, pushing an object that slides after release, and in-hand reorientation. The reorientation task uses the median-rotations metric of Learning Dexterous In-Hand Manipulation, whose vision-based block policy dropped from 33 rotations in simulation to 11.5 in reality. Each policy gets twenty trials per task.
Metrics. We report real success, the gap between simulated and real success, visual encoder calls per second, and end-to-end latency.
Decisive checks. Four outcomes decide the project.
- The core claim fails if the dense student with timing randomization transfers as well as our policy.
- The core claim is supported if our policy's real-world advantage tracks the dense policy's measured sensitivity to temporal rendering perturbations across tasks.
- The secondary claim stands or falls on the matched-rate comparison between contact-triggered and uniform scheduling on the insertion and reorientation tasks. Those are the only tasks where we expect it to matter.
- If our policy matches dense success while running the visual encoder at least five times less often, the method is worthwhile on edge hardware even if the transfer advantage is small.
References
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
- Less Is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
- Adaptive Keyframe Sampling for Long Video Understanding
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection
- KeyWorld: Key Frame Reasoning Enables Effective and Efficient World Models
- Real-Time Execution of Action Chunking Flow Policies
- Learning dexterous in-hand manipulation
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer
- Contact-Aware Neural Dynamics
- Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization
- Robust Visual Sim-to-Real Transfer for Robotic Manipulation
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills