Depth Helps: Improving Pre-trained RGB-based Policy with Depth Information Injection

Xincheng Pang, Wenke Xia, Zhigang Wang, Bin Zhao, Di Hu, Dong Wang, Xuelong Li

TL;DR

This paper introduces the Depth Information Injection (DI²) framework, which enhances pre-trained RGB-based robotic manipulation policies by cleverly incorporating depth information during training. The key innovation is a method that allows the policy to perform well even when only RGB input is available during deployment, by learning to predict depth from RGB.

What Problem Does This Paper Address?

Generalizable robotic manipulation is a major goal in AI, and recent foundation models have shown impressive capabilities using RGB (color) images. However, these models often struggle with fine-grained manipulation tasks because they lack a true understanding of 3D space. Relying solely on RGB limits their ability to perceive depth, shape, and precise spatial relationships, which are critical for tasks like grasping and assembling objects. While using RGB-D (color and depth) data during training can improve performance, deploying these models in real-world scenarios where depth sensors might be unavailable or unreliable is a significant challenge. This paper aims to bridge this gap by enabling robust manipulation policies that are trained with RGB-D but can operate effectively using only RGB during inference.

Methods

The core of the DI² framework lies in two novel modules:

  • Depth Completion Module (DCM): This module is designed to extract spatial prior knowledge from RGB images and generate "virtual" depth information. It uses a Perceiver Resampler to capture modality-specific biases related to depth and integrates learnable tokens that act as queries to estimate depth features from RGB features. Essentially, it learns to "imagine" depth from color.
  • Depth-Aware Codebook (DAC): To combat the cumulative errors that can arise from depth prediction, the DAC discretizes the predicted depth features. It uses a codebook of quantized vectors to represent depth information more robustly, filtering out noise and ensuring more accurate depth estimates. This is inspired by techniques used in vector quantization for generative models.

During training, the framework uses both RGB and depth data. The DCM learns to predict depth features from RGB, and the DAC refines these predictions. For deployment (inference), the policy only receives RGB input, but it leverages the learned DCM to generate depth features, which are then used alongside RGB features to produce manipulation actions.

Datasets

  • Simulated LIBERO environments: The paper uses the LIBERO benchmark for training and evaluating simulated robotic manipulation tasks. LIBERO is described as a large-scale benchmark with 130 tasks, covering object manipulation, spatial reasoning, long-horizon tasks, and goal-oriented scenarios.
  • Real-world scenarios: The authors also conducted experiments in real-world settings, using a Franka Emika Panda robotic arm and two Intel D435 cameras.

Key Insights and Ideas

  • Depth is crucial for fine-grained manipulation: The paper strongly emphasizes that while RGB is powerful, it's insufficient for complex, precise manipulation tasks.
  • Learning to predict depth from RGB is feasible and effective: The DI² framework demonstrates that a policy can learn to infer depth information from RGB alone, enabling it to perform well even without a depth sensor during deployment.
  • Codebooks can improve depth prediction robustness: The Depth-Aware Codebook is shown to be effective in reducing noise and cumulative errors in predicted depth, leading to more reliable action sequences.
  • A staged training approach is beneficial: The paper outlines a warm-up phase, an alignment phase (for DCM), and a codebook training phase, suggesting a structured approach to learning these complex representations.

Key Results

  • Improved performance on LIBERO: The DI² framework achieved the highest overall average success rate (63.95%) on the LIBERO benchmark when trained with RGB-D and tested with RGB-only input. This was a significant improvement over baseline methods.
  • Minimal performance drop in RGB-only: Crucially, the proposed method showed the smallest performance drop (from 63.95% to 63.15%) when switching from RGB-D to RGB-only input, highlighting its robustness.
  • Real-world validation: Experiments in a real-world setting with four complex tasks showed that the DI² framework achieved strong performance using RGB-only input, outperforming other RGB-only baselines. For instance, in the "Close drawer" task, the DI² method achieved 66.67% success with RGB-only, compared to 3.33% for a baseline RGB-D-RF model tested in RGB-only mode.
  • Ablation studies confirm module effectiveness: The DCM was found to significantly improve performance across most task suites, while the DAC primarily boosted performance on the Goal suite.

Limitations and Future Work

  • Limitations: The paper notes that the Depth-Aware Codebook led to a slight performance drop in the "Object" suite, which they attribute to the suite's specific characteristics. The reliance on a pre-trained VLM (Visual-Language Model) is also an implicit limitation, as the performance is tied to the capabilities of the underlying VLM.
  • Future Work: While not explicitly detailed, the success in real-world scenarios suggests potential for broader deployment in robotics. The framework's ability to adapt to different manipulation tasks also opens avenues for exploring more complex and diverse robotic applications. The authors' release of their website and code suggests a commitment to further development and community engagement.

Summary updated

Abstract

3D perception ability is crucial for generalizable robotic manipulation. While recent foundation models have made significant strides in perception and decision-making with RGB-based input, their lack of 3D perception limits their effectiveness in fine-grained robotic manipulation tasks. To address these limitations, we propose a Depth Information Injection (DI2) framework that leverages the RGB-Depth modality for policy fine-tuning, while relying solely on RGB images for robust and efficient deployment. Concretely, we introduce the Depth Completion Module (DCM) to extract the spatial prior knowledge related to depth information and generate virtual depth information from RGB inputs to aid policy deployment. Further, we propose the Depth-Aware Codebook (DAC) to eliminate noise and reduce the cumulative error from the depth prediction. In the inference phase, this framework employs RGB inputs and accurately predicted depth data to generate the manipulation action. We conduct experiments on simulated LIBERO environments and real-world scenarios, and the experiment results prove that our method could effectively enhance the pre-trained RGB-based policy with 3D perception ability for robotic manipulation. The website is released at https://gewu-lab.github.io/DepthHelps-IROS2024.
BibTeX
@inproceedings{pang2024depth,
  author = {Xincheng Pang and Wenke Xia and Zhigang Wang and Bin Zhao and Di Hu and Dong Wang and Xuelong Li},
  title = {{Depth Helps: Improving Pre-trained RGB-based Policy with Depth Information Injection}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802706},
}
Figure