TL;DR
Skill Abstraction from Optical Flow (SOF) enables robots to discover and compose reusable physical skills directly from action-free videos without explicit action labels. By using optical flow as a motion-centric surrogate for action, SOF learns a discrete skill library, plans at the skill level, decodes plans into predicted optical flow, and translates that flow into executable robot actions with minimal labeled demonstration data.
What Problem Does This Paper Address?
While internet and third-person videos capture diverse physical interactions across many environments, translating action-free videos into robot control remains a challenge:
- Lack of actions: Uncurated videos contain pixel data but no underlying joint commands or motor signals.
- Trajectory-level vs. single-step extremes: Prior video-based robot learning often either operates at the single-step level (predicting the immediate next frame or latent action, which lacks long-horizon guidance) or at the full demonstration level (which entangles task-specific visual details and fails to generalize).
- Appearance overfitting: Learning skills directly from raw RGB frames often causes models to overfit to scene backgrounds, textures, or specific embodiments rather than the underlying physics and motion.
SOF targets the sweet spot: discovering structured, modular, and reusable action primitives (e.g., reaching, pushing, grasping) that can be recombined across different environments and tasks.
Methods
SOF uses a three-stage framework:
- Learning Skill Abstractions from Optical Flow:
- Flow as Action Surrogate: Computes optical flow between consecutive video frames using off-the-shelf estimators (FlowFormer++ for real-world video, NeuFlow-v2 for simulation) to focus purely on pixel motion while discarding appearance noise.
- Quantized Skill Autoencoder: Adapts the QueST architecture. A causal convolutional encoder processes optical flow chunks of horizon $H$ and quantizes them into discrete skill tokens using Finite Scalar Quantization (FSQ).
- Position Conditioning: The flow decoder is conditioned on the initial RGB frame to disentangle relative motion primitives from absolute starting positions.
- Skill-Level Planning:
- An autoregressive Transformer skill predictor takes the current RGB observation and a natural language task embedding, then predicts the sequence of discrete skill tokens required to complete the task.
- Action Execution via Flow-to-Action:
- The predicted skill tokens are decoded into predicted future optical flow.
- An execution module converts the optical flow sequence into robot actions. The authors evaluate two variations:
- Learning-Free: Estimates rigid SE(3) transformations from flow and uses heuristic grasping and path following (adapted from AVDC; requires depth and masks).
- Learning-Based (Primary): A lightweight ResNet-18 regression module trained on a very small set of action-labeled trajectories to map flow directly into joint or end-effector actions.
Datasets
The authors evaluate SOF across simulated manipulation benchmarks and one real-world dataset:
- MetaWorld (Simulation): 9 tasks used for multi-task evaluation and 10 tasks for cross-embodiment evaluation, using third-person camera views.
- LIBERO (Simulation): 10 tasks from LIBERO-GOAL for multi-task manipulation, and 4 long-horizon compound tasks from LIBERO-10.
- BridgeData V2 (Real-World): Used for qualitative skill space and clustering analysis of flow plans under complex real-world visual diversity.
(No new benchmark datasets were created.)
Key Insights and Ideas
- Motion over Appearance: Optical flow serves as an effective abstraction barrier. It strips out distracting visual textures, lighting changes, and backgrounds, enabling the model to group identical kinematic motions across visually distinct scenes.
- Flow Outperforms Next-Frame Prediction: Replacing flow with next-frame RGB prediction dropped MetaWorld success by 13%. RGB prediction frequently generated contradictory behaviors (e.g., reversing direction midway or confusing "open faucet" with "close faucet" due to visual symmetry).
- The Value of an Explicit Flow Plan: Directly mapping discrete skill tokens to actions without decoding optical flow led to severe drops in performance (success rates fell from 0.49 to 0.15–0.21). Intermediate flow prediction acts as an essential geometric prior.
- Shared Skills Transfer Across Embodiments: Pretraining skill tokens on both Franka Panda and Sawyer arms produced nearly identical skill-token trajectories for the same task, proving the high-level motion space is embodiment-agnostic even when execution dynamics vary.
Key Results
- Multi-Task Performance (MetaWorld): With 50 action-free videos and only 10 action-labeled demos per task, SOF achieved an overall success rate of 0.69, outperforming Behavior Cloning (0.57), Diffusion Policy (0.31), AVDC (0.42), and latent-action baseline LAPA (0.14).
- Multi-Task Performance (LIBERO-GOAL): SOF achieved 0.25 average success compared to BC (0.23), Diffusion Policy (0.18), and LAPA (0.00).
- Long-Horizon Tasks (LIBERO-10): Given 10 labeled demonstrations, SOF scored 0.16 overall, whereas BC scored 0.03 and Diffusion Policy scored 0.01. BC and Diffusion Policy required 30 demonstrations to match SOF's 10-demo performance.
- Cross-Embodiment Transfer: When policy learning was trained on only one arm's data, policies transferred with minimal drop: Sawyer-trained policies reached 50.7% on Panda (topline 51.6%), and Panda-trained policies reached 55.1% on Sawyer (topline 60.4%).
- Inference Speed: Runs at ~15 Hz end-to-end on a single RTX 4090, which is faster than Diffusion Policy (~10 Hz).
Limitations and Future Work
- Small Object & High-Rotation Failures: Struggles on tasks involving tiny objects (e.g., small bottles or cream containers) or large rotations (e.g., opening drawers), where subtle gripper movements or large occlusions degrade flow quality.
- 2D Viewpoint Dependence: Relies on fixed camera positions and 2D pixel motion; severe arm occlusions can corrupt the inferred action. Extending representations to 3D scene flow is suggested.
- Physical Hardware Deployment: Real-world validation was restricted to offline analysis on BridgeData V2; closing the loop on physical robots remains future work.
Summary updated