Introduction
Robotic imitation learning (IL) has achieved significant milestones in dexterous manipulation, yet these systems often remain confined to controlled laboratory settings. A major hurdle to real-world deployment is "perceptual fragility"—the tendency for robotic policies to fail when environmental conditions, such as lighting or camera exposure, deviate from the training data. Standard RGB-based models often over-rely on color and texture features that are highly sensitive to these shifts, leading to performance collapse in dynamic environments.
Methodology
To overcome visual sensitivity, we propose a framework that integrates monocular depth estimation with a novel data augmentation strategy to provide a stable spatial reference for action prediction.
Monocular Depth Estimation
We utilize a state-of-the-art monocular depth estimation model (Depth Anything V2) to extract 3D spatial information from standard RGB frames. Depth maps serve as a "lighting-invariant anchor," maintaining structural consistency even when RGB frames become unusable due to extreme exposure variations.
AugBlender and Multimodal Fusion
To prevent the model from taking the "easy path" of memorizing unreliable color patterns, we introduce the AugBlender pipeline. This module deliberately applies out-of-distribution (OOD) corruptions—such as extreme hue shifts and solarization—to RGB inputs during training. This adversarial approach forces the policy to prioritize the stable depth information. The corrupted RGB and clean depth maps are fused via a Feature Pyramid Network (FPN) with a ResNet34 backbone, which then informs a transformer-based Diffusion Policy.
Experimental Evaluation
The framework was evaluated across three robotic tasks: CupStack, PickSmall, and PickBig. We curated datasets with both fixed and varied exposures (ranging from 50ms to 160ms) and tested the models across ten discrete exposure levels, from near-darkness (10ms) to extreme brightness (170ms).
Results and Discussion
The proposed framework demonstrated a substantial increase in success rates compared to standard baselines. In the CupStack task, the proposed model achieved an 88% average success rate, significantly outperforming the 23% achieved by the baseline RGB-only Diffusion Policy.
Key findings include:
- Resilience to Extremes: At the lowest exposure level (10ms), the baseline failed completely (0% success), while our model maintained a 62% success rate.
- The Necessity of Spatial Anchors: Simply increasing data diversity through varied exposure training without the depth/AugBlender framework actually decreased performance, suggesting that varied lighting without a stable reference destabilizes the learning process.
- Modality Trust: By providing a "clean" depth map alongside a "noisy" RGB image, the model learns to rely on structural ground truths that lighting cannot obscure.
Conclusion
By leveraging depth estimation as a structural anchor and employing adversarial augmentation via AugBlender, robotic policies can achieve high robustness against environmental visual noise. Future work will address long-horizon spatio-temporal memory and physical occlusions to further enhance deployment capabilities in complex real-world scenarios.
Summary updated