All-For-One: Distilling Visual Foundation Models to a Fast, Memory-Efficient Visuomotor Policy

Byungkun Lee, Minho Park, Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Kyungmin Lee, Yunho Kim, Jaegul ChooarXiv 2026 · arXiv

TL;DR

All-For-One (AFO) is a two-stage distillation framework that allows robots to benefit from the rich semantic knowledge of large Visual Foundation Models (VFMs) without the massive computational overhead. By treating VFMs as "privileged information" during training, AFO produces a lightweight, RGB-only student policy that is 50x faster than its teacher and actually outperforms it by avoiding the temporal jitter and prediction errors inherent in off-the-shelf vision models.

What Problem Does This Paper Address?

Visual Foundation Models like SAM 2 (segmentation) and Depth Anything (depth estimation) are incredibly powerful, but they are notoriously difficult to use for real-time robotic control. The authors identify three main bottlenecks:

  1. Computational Cost: These models are huge. Running them in a closed-loop control system introduces significant latency, making the robot unresponsive.
  2. Temporal Inconsistency: VFMs often process images frame-by-frame, leading to "jitter" in depth or mask predictions that can confuse a controller.
  3. Distribution Shift: Most VFMs are trained on web-scraped images (exocentric), which look very different from the "in-the-trenches" egocentric views a robot sees.

As shown in Figure 1, researchers usually face a choice: train a small model from scratch (fast but inaccurate) or use a frozen VFM backbone (accurate but slow). AFO aims to break this tradeoff.

Figure 1. Overcoming the Compute–Performance Tradeoff. Learning from scratch (LfS) [17] is fast but inaccurate, whereas VFM backbones [35, 37, 39, 43, 50] are accurate but slow. By distilling large-scale pretrained VFMs [10, 46, 58] into a compact student, AFO attains high success rates while remaining computa- tionally efficient at inference.

Methods

The AFO framework operates in two distinct stages, following the "privileged learning" paradigm:

1. The "All" Stage (Teacher Training)

A heavy teacher policy is trained using "privileged" visual modalities. It doesn't just look at RGB; it also sees:

  • Object Segmentation: Generated by SAM 2, which is automatically prompted by a Vision-Language Model (Molmo) to find task-relevant objects (e.g., "pick up the red mug").
  • Depth Estimation: Generated by Depth Anything V2 (DAM 2).

These inputs are fused into a shared latent space using an additive fusion architecture. The teacher learns a high-performance policy using standard Behavior Cloning (BC) on expert demonstrations.

2. The "For-One" Stage (Student Distillation)

This is where the magic happens. A lightweight student model (a simple CNN) is trained to mimic the teacher. Crucially, the student only receives raw RGB images.

  • Feature Alignment: The student is forced to align its internal representations with the teacher’s VFM-enriched features using a cosine similarity loss.
  • Action Imitation: The student also learns to predict the same actions as the expert.

By the end of training, the student has "internalized" the geometric and semantic cues of the VFMs, allowing it to run at high speeds without ever calling a VFM at inference time.

Figure 2. Overview of the All-For-One (AFO) Framework.

Datasets

The authors evaluated AFO across a diverse set of environments:

  • Adroit: High-dimensional dexterous manipulation tasks (Reorient-Pen, Relocate) using a 24-DoF shadow hand.
  • MetaWorld: A standard tabletop manipulation suite (Hammer, Assembly, Bin Picking, etc.).
  • Real-World Tasks: Three custom tasks performed on a 6-DoF Indy7 arm: Clutter Sweeping, Clean Toolbox (a long-horizon task), and Pick-and-Place.

Key Insights and Ideas

  • The Student Surpasses the Teacher: In a surprising twist, the distilled student often performs better than the VFM-heavy teacher. Why? Because the teacher is at the mercy of the VFM's occasional mistakes (like misidentifying a shadow as a solid object or losing track of a mask). The student, trained end-to-end, learns to extract the useful parts of the VFM knowledge while ignoring the noise and jitter.
  • In-Distribution Distillation is Key: Unlike prior work (like Theia) that distills VFM features on generic datasets like ImageNet, AFO distills directly on the robot's task data. This "grounds" the features in actual manipulation, making them much more effective for control.
  • Robustness to Visual Shift: Because the student has learned to focus on task-relevant geometry (thanks to the depth and mask supervision), it is significantly more robust to changes in lighting, background, and clutter than models trained from scratch.
Figure 3. Visual illustration of the simulation and real-world benchmarks.

Key Results

  • Performance: AFO Student achieved a 92.57% average success rate across 7 simulation tasks, outperforming the strongest VFM-based baselines (like R3M and Theia).
  • Efficiency: The student uses only 2.7M parameters and has a latency of 2.11ms on an A100 GPU. For comparison, the teacher policy requires 8.2B parameters and has a latency of 110.07ms—a 50x speedup.
  • Real-World OOD: In out-of-distribution real-world tests (changing desk mats, lighting, and adding clutter), AFO maintained a 52.59% success rate, while the standard ACT baseline dropped to 43.33%.

Limitations and Future Work

The authors note that while their lightweight CNN encoders are excellent for real-time deployment, they may hit a performance ceiling compared to massive Vision-Language-Action (VLA) models. A promising future direction is applying this "privileged VFM distillation" to VLAs to retain their generalization capabilities while slashing their massive inference costs. Additionally, the current method treats all VFM cues as equally important; a task-adaptive weighting system could further optimize performance for specific scenarios.

Summary updated

Abstract

Visual Foundation Models (VFMs) provide semantic cues such as depth and segmentation useful for robotic manipulation, but directly deploying them in closed-loop control is impractical due to high computational cost, temporal inconsistency, and distribution shift between static internet images and egocentric robot observations. We propose All-For-One (AFO), a two-stage distillation framework that leverages VFMs strictly as privileged information during training, using downstream robotic data. Following the privileged learning paradigm, AFO decouples semantic representation learning from inference-time policy execution. In the All stage, a teacher policy fuses RGB observations with outputs from multiple VFMs to learn a semantically enriched representation. In the For-One stage, this knowledge is distilled in-distribution into a lightweight RGB-only student policy. Evaluated on two Adroit tasks and five MetaWorld tasks in simulation, as well as three real-world manipulation tasks, AFO outperforms prior foundation-model-based approaches while using over 30x fewer parameters than VFM-based encoders and 50x lower latency than its own teacher in simulation benchmarks, providing an efficient pathway for transferring foundation-model knowledge to robotic control.
BibTeX
@misc{lee2026all,
  author = {Byungkun Lee and Minho Park and Hojoon Lee and Dongyoon Hwang and Donghu Kim and Hyunseung Kim and Kyungmin Lee and Yunho Kim and Jaegul Choo},
  title = {{All-For-One: Distilling Visual Foundation Models to a Fast, Memory-Efficient Visuomotor Policy}},
  year = {2026},
  eprint = {2611.04485},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04485},
}
Figure