Robots that must react to fast-moving objects, such as catching a ball or dodging a rolling obstacle, need to perceive motion and not just appearance. A practical recipe has emerged in simulation. A large optical-flow network serves as the "teacher": it estimates the per-pixel velocity field between consecutive frames. Its output is compressed into a tiny "student" network fast enough to run inside a control loop, and a reinforcement-learning policy is trained on the student's motion estimates. This recipe solves dynamic tasks where frame-stacking agents fail. It has been validated only inside the simulator, however, and the student was trained on simulator renderings. Nobody knows whether the motion features survive real cameras, with their blur, rolling shutter, lighting changes, and non-rigid objects.
We propose to treat the flow student as the place where the gap between simulation and reality should be closed, and to close it with real footage instead of real reinforcement learning. The policy is trained entirely in simulation and sees only motion fields. The student that produces those fields is distilled on a mix of simulator frames, supervised by exact simulator flow, and unlabeled real video from the robot's own camera, supervised by teachers that learned motion from real-world video. A temporal-consistency term keeps the student's frame-to-frame noise from turning into jittery motor commands. The claim is testable on hardware. If the gap mostly lives in the motion module, this change alone should recover most of the policy's simulated performance on a physical catching task, with zero real-world policy training.
Core Claim
For a policy that consumes only predicted motion fields, the transfer gap is dominated by the mismatch between the student's motion estimates on real images and on simulated images, not by the policy itself. Distilling the student on unlabeled real footage from real-video-trained teachers, with an explicit penalty on temporally incoherent flow, should therefore transfer a simulation-trained interception policy to a physical robot better than four alternatives: rendering randomization alone, running a full flow network on board, or frame-stacking policies with action smoothing. A secondary claim concerns evaluation. The student's flow error and temporal jitter on replayed real footage, which are cheap offline measurements, should predict physical success rate. That would give motion-distilled policies a real-world validation signal that does not require running the robot for every design choice.
Where This Sits in Prior Work
Two papers motivate the gap directly. Motion Perception in Reinforcement Learning with Dynamic Objects distills FlowNet 2.0 into TinyFlowNet on image pairs rendered by the simulator. It shows that flow-augmented agents solve 3D Catcher and 3D KeepUp, where frame-stacking agents collapse into "lazy" stationary behavior. The authors name physical testing under real lighting and blur as unaddressed future work. End-to-End Interactive Prediction and Planning With Optical Flow Distillation for Autonomous Driving distills flow-aware bird's-eye-view features into a flow-free driving student, which reaches 91.28% success in CARLA at 0.06 s per frame. Its evaluation stops at CARLA and logged nuScenes data. We keep both papers' architectures, the small flow student feeding a policy. We change the data the student is distilled on and the place where it is evaluated.
Two papers that did cross to hardware constrain the design. Privileged Knowledge Distillation for Sim-to-Real Policy Generalization transfers a quadruped zero-shot by compressing proprioceptive history toward simulator-only physical parameters. Its privileged signal is a handful of scalars, so it says nothing about high-dimensional visual motion. We borrow its lesson that what the student is asked to reconstruct matters more than how faithfully it imitates. Learning from Simulation, Racing in Reality finds that unregularized simulation policies crash on a physical race car because their actions chatter. Commanding action rates, plus an output penalty, fixes this. We adopt that action parameterization in every arm, so that our gains cannot come from smoothing alone. Distill, Suppress, and Fuse shows that flow-to-RGB distillation transfers high-frequency noise that hurts segment-level accuracy even when per-frame accuracy rises. That is the perception-side analogue of chattering, and it motivates our temporal-consistency term.
The teachers come from two papers on extracting motion from real video. Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals and Taming generative video models for zero-shot optical flow extraction probe video models pretrained on internet video. They obtain flow that beats synthetic-supervised networks on real non-rigid motion: 47.53 and 44.16 Average Jaccard on TAP-Vid DAVIS respectively. Both note that their probing is far too slow for online control, and neither deploys a distilled student in a closed loop. We use their slowness where it does not matter, labeling recorded footage offline, and deploy only the student. As a cheaper real-video teacher we also use RAFT, a recurrent flow network trained on synthetic data and fine-tunable on real benchmarks.
Research Program
The task is a physical analogue of the Catcher environments: a tabletop arm with a single onboard RGB camera intercepts balls rolled or tossed across the workspace. We build a matching simulated scene. The policy is trained with Proximal Policy Optimization, a standard on-policy reinforcement-learning algorithm. It sees the current frame plus student-predicted flow and outputs joint-velocity rates, following Chisari et al.
The student is a TinyFlowNet-sized network, fixed across all arms, distilled with a three-part loss. On simulator frames it regresses exact rendered flow. On real frames it regresses teacher pseudo-labels, weighted down where the teacher is uncertain. On consecutive real triples it penalizes changes in predicted flow that the teacher's own flow does not exhibit:
$$\mathcal{L} = \lVert s(x^{sim}) - f^{sim} \rVert_1 + \lambda_r\, w \odot \lVert s(x^{real}) - f^{T} \rVert_1 + \lambda_c \lVert (s_t - s_{t-1}) - (f^{T}_t - f^{T}_{t-1}) \rVert_1 .$$
Here $s(\cdot)$ is the student's flow, $f^{sim}$ is simulator flow, $f^{T}$ is teacher flow on real video, $w$ is a per-pixel confidence mask from teacher forward-backward consistency, subscripts index time, and $\lambda_r, \lambda_c$ are weights. The real footage is about ten hours of unlabeled video from the robot camera: balls, hands, clutter, and varied lighting, with no task labels.
Crucially, the policy is trained in simulation on the student's flow on simulated frames, not on ground-truth flow. Whatever the student does on real images should then resemble what the policy already saw. We will analyze where the gap lives by measuring the distance between the student's flow statistics on real and on simulated clips of matched motion, and relating it to physical success.
Baselines
The naive reference is the setup of Amiranashvili et al. as published: TinyFlowNet distilled from FlowNet 2.0 on simulator renderings only, with the policy trained on its outputs. The only change is our action-rate parameterization. This is the "do nothing about reality" point. The file gives no numeric rewards for it, so its physical success rate will be measured here. A second weak reference is a frame-stacking policy with the same action smoothing, which tests whether explicit motion is needed at all on hardware.
The strongest competitor is running a full real-video-capable flow network on board in place of the student. Concretely, this is RAFT fine-tuned on the same real footage, used at the lowest resolution that keeps the control rate. RAFT has 5.3M parameters and runs at 10 FPS at 1088×436 on a GTX 1080Ti. No closed-loop robotic result for it was found on Lacuna. It is the strongest because it has no distillation loss at all. The policy for this arm is retrained in simulation on RAFT's simulated-frame flow, and it gets the same training budget and hardware.
The second strong competitor is aggressive rendering randomization of textures, lighting, blur, and noise when distilling the student in simulation, with no real footage. It is tuned with the same hyperparameter budget as our method.
Ablations isolate each ingredient: our method without the consistency term ($\lambda_c=0$); with simulator data only versus real data only; with RAFT versus Opt-CWM versus KL-tracing as the real-video teacher; and with the policy trained on ground-truth simulator flow instead of student flow.
Beating the field means a higher physical interception rate than both strong competitors over at least 100 throws per arm, with confidence intervals that do not overlap. It must also come with lower measured joint jerk than the variant without the consistency term, at a control rate the on-board flow network cannot match.
Why not just…?
Why not just run a fast flow network on board? This is the most serious threat. At small resolutions a modern flow network may be fast enough, and it avoids distillation noise. The answer cannot be argued in advance. The on-board RAFT arm above settles it: we report success as a function of control rate, so we can see whether the student wins on latency or the full network wins on accuracy. The latency figures in the file, 0.21 s versus 0.06 s for the INMP teacher and student, show the gap is often large. Interception at high ball speeds is exactly where latency should dominate. If RAFT matches us at equal rate, the paper's contribution narrows to the evaluation finding, and we will say so.
Why not just randomize rendering when distilling in simulation? Randomization only covers variation the simulator can express. The failures documented by Opt-CWM and KL-tracing involve non-rigid motion, reflections, and real blur, which synthetic engines model poorly. The rendering-randomization baseline measures how much of the gap this closes. Our flow-statistics analysis shows whether its residual error sits in phenomena that are absent from the simulator.
Why not just smooth actions and fine-tune the policy on real data? Chisari et al. needed three hours of real driving to approach the model-predictive controller after transfer, and every new scene would need that again. Our arms all use their action smoothing. Smoothing therefore cannot explain any difference between them, and real-world policy fine-tuning remains an orthogonal addition rather than a competitor.
Why not just use simulator ground-truth flow as the policy input? It is the perfect teacher in simulation. It also guarantees that the policy has never seen the student's errors. The ground-truth-flow ablation tests this directly.
Proposed Experiments and Decisive Checks
Offline, we measure student quality on real footage in three ways. We report TAP-Vid DAVIS Average Jaccard from chained flow, with Opt-CWM's 47.53 as a ceiling reference for the teacher rather than a target for a tiny student. We report KITTI-2015 F1-all. We also report a jitter score, the high-frequency energy of flow on static-camera real clips. On the robot, we report interception rate across three ball speeds and two lighting conditions, joint jerk, and control rate.
The decisive check has two parts. The first is whether real-footage distillation recovers most of the simulation-to-hardware drop in interception rate relative to the naive reference and rendering randomization. The second is whether offline real-replay flow error and jitter rank the twelve-odd arms in the same order as physical success. A rank correlation near zero would mean the offline proxy is useless, and we would report that as a negative result about simulation-plus-replay evaluation. A small simulation-to-hardware gap for the naive reference would refute the premise that the motion module is the bottleneck, and would point back toward actuation and latency, as Chisari et al. found.
References
- Motion Perception in Reinforcement Learning with Dynamic Objects
- End-to-End Interactive Prediction and Planning With Optical Flow Distillation for Autonomous Driving
- Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective
- Learning from Simulation, Racing in Reality
- Distill, Suppress, and Fuse: Cross-Modal Knowledge Integration for Optical Flow-Free Temporal Action Segmentation
- Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals
- Taming generative video models for zero-shot optical flow extraction
- RAFT: Recurrent All-Pairs Field Transforms for Optical Flow