Depth Helps: Improving Pre-trained RGB-based Policy with Depth Information Injection

Xincheng Pang, Wenke Xia, Zhigang Wang, Bin Zhao, Di Hu, Dong Wang, Xuelong Li

Abstract

Robotic policies often rely on RGB input, which lacks explicit 3D spatial information, leading to suboptimal performance in fine-grained manipulation tasks. While RGB-D sensors provide depth, they are frequently unavailable or noisy in real-world deployments. This paper introduces the Depth Information Injection ($DI^2$) framework, which enables RGB-based policies to "imagine" 3D spatial features at inference time. $DI^2$ utilizes a Depth Completion Module (DCM) to extract latent depth features from RGB images and a Depth-Aware Codebook (DAC) to discretize these features, mitigating accumulative errors. Evaluated on the LIBERO benchmark and real-world robotic tasks, $DI^2$ significantly improves success rates, particularly in tasks requiring high spatial precision, while maintaining stability without physical depth sensors.

Introduction

Most modern robotic "foundation models" rely on RGB (color) images to make decisions. While RGB is effective for object identification, it often fails to provide the precise 3D spatial awareness required for fine-grained tasks like stacking blocks or inserting a key. Although adding a depth sensor (RGB-D) can solve this, such sensors are often noisy, power-hungry, or unavailable in deployment scenarios.

Previous attempts to bridge this gap, such as distilling knowledge from a depth-aware teacher model to an RGB student, frequently suffer from sequential accumulative errors. In these cases, minor depth prediction errors at the start of a trajectory lead to catastrophic failures over time. The $DI^2$ framework addresses this by giving RGB-only robots a stable 3D "intuition" that remains accurate throughout long-horizon tasks.

Methodology

The $DI^2$ framework introduces two novel modules to a standard Visual Language Model (VLM) policy to inject spatial awareness without requiring depth sensors at deployment.

Depth Completion Module (DCM)

Instead of reconstructing a full depth image, the DCM predicts latent depth features. It utilizes a set of "learnable tokens" (spatial priors) and a Perceiver Resampler to extract 3D information hidden within the RGB input. This approach bypasses the computational overhead of image reconstruction while capturing the essential spatial geometry needed for manipulation.

Depth-Aware Codebook (DAC)

The DAC serves as the framework's stabilizer. To prevent the accumulative errors common in predicted-modality models, the DAC maps the continuous depth features from the DCM to the nearest discrete vectors in a pre-trained codebook. By "rounding" noisy features to fixed "depth concepts," the DAC filters out modality-specific noise and ensures the policy receives a clean, consistent signal.

The $DI^2$ framework architecture, showing the integration of the DCM and DAC modules within a VLM policy to process RGB and Language inputs.

The training process is divided into three stages: a Warm-up phase (learning the task with real depth data), an Align phase (teaching the DCM to mimic depth features using only RGB), and a Codebook phase (optimizing the discrete feature mapping).

Experimental Evaluation

The researchers validated $DI^2$ using both simulation and real-world platforms:

  • LIBERO Benchmark: A large-scale simulation containing 130 manipulation tasks across four suites: Object (interaction), Spatial (positioning), LongHorizon (multi-step), and Goal (vision-based targets).
  • Real-World Scenarios: Deployment on a Franka Emika Panda robot arm for four tasks: closing a drawer, picking up a banana, cooking a pepper, and stacking a block.

Results and Discussion

The $DI^2$ framework consistently outperformed RGB-only baselines and other cross-modal distillation methods.

Simulation Performance

In the LIBERO benchmark, the method achieved a 63.15% average success rate using only RGB at inference, compared to 57.95% for the standard RGB-only baseline (RoboFlamingo). Remarkably, the performance drop when switching from real RGB-D data to "imagined" depth via $DI^2$ was only 0.8% (63.95% to 63.15%), whereas other methods saw significant performance degradation.

Real-World Precision

In the "Stack the block" task—which requires high 3D precision—the $DI^2$ model achieved a 73.33% success rate, while the standard RGB baseline struggled at just 20%. This highlights the framework's ability to provide the spatial intuition necessary for precision-heavy tasks.

Qualitative results of real-world robotic tasks, including closing a drawer, stacking blocks, cooking a pepper, and picking up a banana.

Conclusion and Limitations

The $DI^2$ framework demonstrates that depth information can be effectively injected into RGB-based policies to improve 3D spatial awareness without specialized hardware at deployment. While the Codebook (DAC) significantly improves stability, the authors noted it slightly reduced performance in tasks requiring extremely fine-grained object identification, likely due to the quantization process "rounding off" subtle geometric details. Future work may explore adaptive codebooks or scaling the framework to more diverse "in-the-wild" datasets where depth data is scarce.

Summary updated

BibTeX
@inproceedings{pang2024depth,
  author = {Xincheng Pang and Wenke Xia and Zhigang Wang and Bin Zhao and Di Hu and Dong Wang and Xuelong Li},
  title = {{Depth Helps: Improving Pre-trained RGB-based Policy with Depth Information Injection}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  doi = {10.1109/iros58592.2024.10802706},
}

Loading related content...

Figure