Lift, See, Act: Hierarchical Robot Policy Pretraining with 3D Foundation Models

Yiyuan Ge, Changxing Ding, Ziyu Hao, Zijie Zheng, Xiangmin XuarXiv 2026 · arXiv

TL;DR

LSA (Lift, See, Act) is a hierarchical pretraining framework that transforms unlabeled human videos into a rich training ground for robots. By distilling dense geometric features and interactive attention patterns from 3D Foundation Models into a visual encoder, LSA enables robots to perform complex 3D manipulation tasks with significantly higher success rates and better robustness than previous 2D-centric methods.

What Problem Does This Paper Address?

Training robots to perform complex tasks is notoriously data-hungry. While human videos are a vast, "in-the-wild" resource, they lack the explicit action labels (like motor torques or 3D coordinates) that robots need to learn.

Existing methods typically try to bridge this gap by extracting 2D visual primitives (like bounding boxes) or sparse 3D hand trajectories. However, these approaches leave a lot of "knowledge" on the table. When a 3D Foundation Model (3D FM) estimates a hand's position, it generates rich, dense intermediate representations of the scene's geometry and depth. Previous work ignored these intermediate signals, treating 3D FMs as "black boxes" that only output a few coordinates. The authors of LSA argue that these dense, internal features are exactly what a robot needs to truly understand the 3D structure of its environment.

Methods

The LSA framework is divided into three synergistic phases:

  1. Lift (3D Representation Elevation): Instead of just using the final output of 3D FMs, LSA aligns the shallow layers of the robot's policy encoder (a Vision Transformer) with the intermediate features of multiple 3D FMs (specifically VGGT for geometric features and Depth Pro for metric depth). A dynamic weight assignment mechanism adaptively balances these "teachers" based on the scene context.
  2. See (Dual Attentive and Structural Guidance): This phase makes the implicit 3D features explicit.
  3. Structural Guidance: The model is tasked with reconstructing a depth map from its internal tokens, forcing it to comprehend spatial layouts.
  4. Attentive Guidance: Using human hand keypoints as a prior, the model is supervised to focus its attention on "interaction hotspots"—the regions where hands meet objects.
  5. Act (Motion Planning): Finally, the model is trained to predict future 3D wrist trajectories. This grounds the learned visual representations in the robot's action space, providing a functional "proxy" for downstream control.

The architecture uses LoRA (Low-Rank Adaptation) to efficiently fine-tune the visual backbone, ensuring the model retains general visual knowledge while specializing in 3D manipulation.

Datasets

  • Pretraining: The primary source is DexYCB, a large-scale dataset of human grasps. The authors also validated the approach's scalability using H2O and Ego4D.
  • Simulation Evaluation: LSA was tested across three major benchmarks:
  • MetaWorld: Tabletop manipulation (e.g., assembly, hammer, dial turn).
  • DexArt: Articulated object manipulation (e.g., opening a toilet lid).
  • Adroit: High-DOF dexterous hand tasks (e.g., pen spinning, door opening).
  • Real-World: Evaluated on a 7-DoF Franka Research 3 robot performing daily tasks like picking trash and wiping a laptop.

Key Insights and Ideas

  • Don't Waste the "Teacher's" Brain: The most surprising insight is that the intermediate features of 3D FMs are more valuable for pretraining than the final 3D coordinates. These dense features provide a "geometric scaffolding" that helps the robot's encoder learn 3D priors from 2D images.
  • Emergent Object Attention: By supervising the model to look at human hands (Attentive Guidance), the authors found that the model naturally learned to attend to the objects being manipulated as well. This cross-domain transfer—from human hands to robot grippers—is key to LSA's success.
  • Robustness through Geometry: Because LSA learns the underlying 3D structure rather than just memorizing 2D textures, it is remarkably robust. It maintains high performance even when the lighting is dimmed or the scene is cluttered with visual "disturbances" that typically cause 2D-based policies to fail.

Key Results

  • SOTA Performance: In MetaWorld simulations, LSA achieved a 92.2% average success rate, outperforming strong baselines like DINOv3 (85.0%) and R3M (26.7%).
  • Real-World Superiority: On a physical Franka robot, LSA reached a 68.3% success rate, nearly 27% higher than the best-performing baseline (HRP).
  • Out-of-Distribution (OOD) Generalization: In real-world tests with unseen towel colors and lighting conditions, LSA showed significantly less performance degradation than competitors, proving it learns generalizable features rather than superficial patterns.
  • Efficiency: Despite its complexity during training, LSA is efficient at inference, running at 230.6 FPS on a single RTX 4090.

Limitations and Future Work

  • The Embodiment Gap: While wrist trajectories are a good proxy, there is still a structural mismatch between human fingers and robot grippers. The authors noted that supervising with full 3D hand meshes actually decreased performance, likely due to this gap.
  • Coarse Spatial Guidance: The current attentive guidance is region-based. Future work could explore finer-grained supervision, such as specific contact points or "affordance" maps.
  • Domain Shift: There is still a visual gap between "in-the-wild" human videos and clean laboratory robot setups. The authors suggest that joint pretraining on both human and robot data could further bridge this divide.

For more details, check out the project page: https://yiyuan00.github.io/Lift-See-Act

Summary updated

Abstract

Robot policy pretraining based on human videos is crucial for improving the policy's generalization ability. One main challenge for this task is the lack of explicit action-relevant representations in such unlabeled data. Recent works tend to estimate 3D hand motion trajectories from videos using 3D foundation models (3D FMs). However, they keep solely the sparse trajectories as supervision, ignoring the rich, fine-grained information produced during the trajectory extraction process. To address this issue, we propose LSA, a hierarchical robot policy pretraining framework that includes three phases: Lift, See, and Act. Specifically, the Lift phase elevates 2D video observations into dense 3D representations by adaptively aligning features in the policy's shallow layers with the intermediate representations extracted from multiple 3D FMs. Building upon these lifted representations, the See phase equips the policy with explicit geometric and interactive awareness through dual guidance. It introduces depth-map reconstruction to help the model comprehend 3D spatial layouts and utilizes hand region cues to explicitly supervise the encoder's attention toward task-relevant interaction hotspots. Empowered by the dense 3D representation learning and precise spatial guidance, our policy achieves robust and accurate robotic manipulation in the Act phase. Extensive experiments on diverse simulation and real-world tasks demonstrate that LSA significantly outperforms current state-of-the-art approaches. The project page is available at: https://yiyuan00.github.io/Lift-See-Act.
BibTeX
@misc{ge2026lift,
  author = {Yiyuan Ge and Changxing Ding and Ziyu Hao and Zijie Zheng and Xiangmin Xu},
  title = {{Lift, See, Act: Hierarchical Robot Policy Pretraining with 3D Foundation Models}},
  year = {2026},
  eprint = {2611.20481},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20481},
}
Figure