TL;DR
All-For-One (AFO) is a two-stage distillation framework that allows robots to benefit from the rich semantic knowledge of large Visual Foundation Models (VFMs) without the massive computational overhead. By treating VFMs as "privileged information" during training, AFO produces a lightweight, RGB-only student policy that is 50x faster than its teacher and actually outperforms it by avoiding the temporal jitter and prediction errors inherent in off-the-shelf vision models.
What Problem Does This Paper Address?
Visual Foundation Models like SAM 2 (segmentation) and Depth Anything (depth estimation) are incredibly powerful, but they are notoriously difficult to use for real-time robotic control. The authors identify three main bottlenecks:
- Computational Cost: These models are huge. Running them in a closed-loop control system introduces significant latency, making the robot unresponsive.
- Temporal Inconsistency: VFMs often process images frame-by-frame, leading to "jitter" in depth or mask predictions that can confuse a controller.
- Distribution Shift: Most VFMs are trained on web-scraped images (exocentric), which look very different from the "in-the-trenches" egocentric views a robot sees.
As shown in Figure 1, researchers usually face a choice: train a small model from scratch (fast but inaccurate) or use a frozen VFM backbone (accurate but slow). AFO aims to break this tradeoff.
Methods
The AFO framework operates in two distinct stages, following the "privileged learning" paradigm:
1. The "All" Stage (Teacher Training)
A heavy teacher policy is trained using "privileged" visual modalities. It doesn't just look at RGB; it also sees:
- Object Segmentation: Generated by SAM 2, which is automatically prompted by a Vision-Language Model (Molmo) to find task-relevant objects (e.g., "pick up the red mug").
- Depth Estimation: Generated by Depth Anything V2 (DAM 2).
These inputs are fused into a shared latent space using an additive fusion architecture. The teacher learns a high-performance policy using standard Behavior Cloning (BC) on expert demonstrations.
2. The "For-One" Stage (Student Distillation)
This is where the magic happens. A lightweight student model (a simple CNN) is trained to mimic the teacher. Crucially, the student only receives raw RGB images.
- Feature Alignment: The student is forced to align its internal representations with the teacher’s VFM-enriched features using a cosine similarity loss.
- Action Imitation: The student also learns to predict the same actions as the expert.
By the end of training, the student has "internalized" the geometric and semantic cues of the VFMs, allowing it to run at high speeds without ever calling a VFM at inference time.
Datasets
The authors evaluated AFO across a diverse set of environments:
- Adroit: High-dimensional dexterous manipulation tasks (Reorient-Pen, Relocate) using a 24-DoF shadow hand.
- MetaWorld: A standard tabletop manipulation suite (Hammer, Assembly, Bin Picking, etc.).
- Real-World Tasks: Three custom tasks performed on a 6-DoF Indy7 arm: Clutter Sweeping, Clean Toolbox (a long-horizon task), and Pick-and-Place.
Key Insights and Ideas
- The Student Surpasses the Teacher: In a surprising twist, the distilled student often performs better than the VFM-heavy teacher. Why? Because the teacher is at the mercy of the VFM's occasional mistakes (like misidentifying a shadow as a solid object or losing track of a mask). The student, trained end-to-end, learns to extract the useful parts of the VFM knowledge while ignoring the noise and jitter.
- In-Distribution Distillation is Key: Unlike prior work (like Theia) that distills VFM features on generic datasets like ImageNet, AFO distills directly on the robot's task data. This "grounds" the features in actual manipulation, making them much more effective for control.
- Robustness to Visual Shift: Because the student has learned to focus on task-relevant geometry (thanks to the depth and mask supervision), it is significantly more robust to changes in lighting, background, and clutter than models trained from scratch.
Key Results
- Performance: AFO Student achieved a 92.57% average success rate across 7 simulation tasks, outperforming the strongest VFM-based baselines (like R3M and Theia).
- Efficiency: The student uses only 2.7M parameters and has a latency of 2.11ms on an A100 GPU. For comparison, the teacher policy requires 8.2B parameters and has a latency of 110.07ms—a 50x speedup.
- Real-World OOD: In out-of-distribution real-world tests (changing desk mats, lighting, and adding clutter), AFO maintained a 52.59% success rate, while the standard ACT baseline dropped to 43.33%.
Limitations and Future Work
The authors note that while their lightweight CNN encoders are excellent for real-time deployment, they may hit a performance ceiling compared to massive Vision-Language-Action (VLA) models. A promising future direction is applying this "privileged VFM distillation" to VLAs to retain their generalization capabilities while slashing their massive inference costs. Additionally, the current method treats all VFM cues as equally important; a task-adaptive weighting system could further optimize performance for specific scenarios.
Summary updated