TL;DR
This paper introduces M³3D, a novel self-supervised pre-training framework that leverages both RGB and depth data to learn rich 3D priors for vision tasks. By combining masked image modeling with cross-modal contrastive learning, M³3D significantly improves performance on various downstream tasks like semantic segmentation and action recognition, especially in low-data regimes.
What Problem Does This Paper Address?
Modern computer vision models, particularly Vision Transformers (ViTs), have achieved remarkable success through self-supervised learning, often using masked autoencoders (MAE) to reconstruct missing parts of images or videos. However, these approaches typically rely on single-modality data. This overlooks the valuable geometric information present in RGB-D (color and depth) sensors, which can provide crucial 3D priors and reduce view-dependent biases. While some methods have explored RGB-D data, they often require camera pose registration or are limited to specific tasks. This paper aims to bridge this gap by developing a more general and effective self-supervised pre-training method that integrates 3D priors from RGB-D data into ViTs for a wider range of 2D image and video understanding tasks.
Methods
M³3D builds upon the Masked Autoencoder (MAE) paradigm but extends it to multi-modal RGB-D inputs. The core idea is to:
- Multi-modal Masked Image Modeling (MIM): Randomly mask patches in both RGB and depth modalities. An encoder (either modality-specific or shared) processes the unmasked patches, and a decoder reconstructs the masked regions. This forces the model to learn robust representations that capture both visual and geometric information.
- Cross-Modal Representation Learning: To further enhance the correspondence between RGB and depth, M³3D incorporates two strategies:
- Contrastive Learning: Inspired by InfoNCE loss, this aims to align features from corresponding RGB and depth patches while pushing dissimilar patches apart. This helps create modality-invariant features.
- Matching Loss: Similar to video-text matching, this loss predicts whether an RGB-depth pair is matched or not, acting as a binary classifier to reinforce cross-modal understanding.
The pre-training can be done in one or two stages. For video tasks, a single-stage approach is used. For image tasks, a two-stage approach is employed: first, contrastive learning for a few epochs, followed by masked image modeling with the encoder initialized from the first stage.
Datasets
The paper uses the following datasets:
- Pre-training:
- ScanNet: A large-scale dataset of RGB-D frames from indoor scenes, used for image-based pre-training.
- UCF-101: A benchmark dataset for video action recognition, used for video-based pre-training.
- OR-AR: A dataset for surgical video action detection, used for video-based pre-training.
- Fine-tuning/Evaluation:
- ScanNet: Used for 2D semantic segmentation.
- NYUv2: Used for depth estimation and 2D semantic segmentation.
- UCF-101: Used for video action recognition.
- OR-AR: Used for surgical video action detection.
Key Insights and Ideas
- 3D Priors are Crucial for 2D Tasks: The paper strongly suggests that incorporating explicit 3D geometric information, even from a single RGB-D view, significantly boosts performance on 2D tasks like semantic segmentation, going beyond what standard MAE can achieve.
- Cross-Modal Learning is Key: Simply masking and reconstructing isn't enough. The contrastive and matching losses are vital for learning truly aligned and complementary representations between RGB and depth, leading to better transferability.
- Data Efficiency: M³3D demonstrates superior data efficiency, achieving strong results even with limited labeled data during fine-tuning, which is a significant advantage in many real-world scenarios.
- Generalizability: The framework is shown to be versatile, effectively transferring learned representations to diverse tasks (action recognition, detection, segmentation, depth estimation) and datasets, including out-of-domain ones.
Key Results
- ScanNet Semantic Segmentation: M³3D achieved a mean Intersection over Union (mIoU) of 67.3%, outperforming Mask3D (+1.3 mIoU) and MAE (+2.8 mIoU).
- NYUv2 Semantic Segmentation: M³3D achieved an mIoU of 51.2%, outperforming Mask3D and other state-of-the-art methods.
- NYUv2 Depth Estimation: M³3D achieved an 81% accuracy (for pixels with error ratio below 1.25), outperforming CroCo (85.6% vs 86.7%) and Mask3D.
- UCF-101 Video Action Recognition: M³3D achieved 96.3% top-1 accuracy, outperforming VideoMAE.
- OR-AR Surgical Video Action Detection: M³3D achieved 80.90 mAP with only 5% labeled data, demonstrating significant data efficiency compared to VideoMAE.
- Data Efficiency on ScanNet: M³3D recovered over 80% of the full labeled performance when fine-tuned with only 20% of the training data.
Limitations and Future Work
- Depth Data Requirement: The primary limitation is the reliance on RGB-D data for pre-training. While the paper shows effectiveness, it's not directly applicable to RGB-only datasets without adaptation.
- Future Work: The authors suggest extending the approach to applications with datasets that have more than two modalities, indicating a path towards even more comprehensive multi-modal learning. They also hypothesize that performance could be further enhanced with improved depth estimation models or readily available high-quality depth data.
Summary updated