Spatially Visual Perception for End-to-End Robotic Learning

Travis Davies, Jiahuan Yan, Xiang Chen, Yu Tian, Yueting Zhuang, Yiqi Huang, Luhui HuarXiv 2024 · DOI · arXiv

Introduction

Robotic imitation learning (IL) has achieved significant milestones in dexterous manipulation, yet these systems often remain confined to controlled laboratory settings. A major hurdle to real-world deployment is "perceptual fragility"—the tendency for robotic policies to fail when environmental conditions, such as lighting or camera exposure, deviate from the training data. Standard RGB-based models often over-rely on color and texture features that are highly sensitive to these shifts, leading to performance collapse in dynamic environments.

Methodology

To overcome visual sensitivity, we propose a framework that integrates monocular depth estimation with a novel data augmentation strategy to provide a stable spatial reference for action prediction.

Monocular Depth Estimation

We utilize a state-of-the-art monocular depth estimation model (Depth Anything V2) to extract 3D spatial information from standard RGB frames. Depth maps serve as a "lighting-invariant anchor," maintaining structural consistency even when RGB frames become unusable due to extreme exposure variations.

Comparison of RGB and depth maps under exposure variations

AugBlender and Multimodal Fusion

To prevent the model from taking the "easy path" of memorizing unreliable color patterns, we introduce the AugBlender pipeline. This module deliberately applies out-of-distribution (OOD) corruptions—such as extreme hue shifts and solarization—to RGB inputs during training. This adversarial approach forces the policy to prioritize the stable depth information. The corrupted RGB and clean depth maps are fused via a Feature Pyramid Network (FPN) with a ResNet34 backbone, which then informs a transformer-based Diffusion Policy.

Overview of the proposed perception model architecture

Experimental Evaluation

The framework was evaluated across three robotic tasks: CupStack, PickSmall, and PickBig. We curated datasets with both fixed and varied exposures (ranging from 50ms to 160ms) and tested the models across ten discrete exposure levels, from near-darkness (10ms) to extreme brightness (170ms).

Results and Discussion

The proposed framework demonstrated a substantial increase in success rates compared to standard baselines. In the CupStack task, the proposed model achieved an 88% average success rate, significantly outperforming the 23% achieved by the baseline RGB-only Diffusion Policy.

Key findings include:

  • Resilience to Extremes: At the lowest exposure level (10ms), the baseline failed completely (0% success), while our model maintained a 62% success rate.
  • The Necessity of Spatial Anchors: Simply increasing data diversity through varied exposure training without the depth/AugBlender framework actually decreased performance, suggesting that varied lighting without a stable reference destabilizes the learning process.
  • Modality Trust: By providing a "clean" depth map alongside a "noisy" RGB image, the model learns to rely on structural ground truths that lighting cannot obscure.

Conclusion

By leveraging depth estimation as a structural anchor and employing adversarial augmentation via AugBlender, robotic policies can achieve high robustness against environmental visual noise. Future work will address long-horizon spatio-temporal memory and physical occlusions to further enhance deployment capabilities in complex real-world scenarios.

Summary updated

BibTeX
@misc{davies2024spatially,
  author = {Travis Davies and Jiahuan Yan and Xiang Chen and Yu Tian and Yueting Zhuang and Yiqi Huang and Luhui Hu},
  title = {{Spatially Visual Perception for End-to-End Robotic Learning}},
  year = {2024},
  eprint = {2411.17458},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.17458},
}
Figure