M33D: Learning 3D Priors Using Multi-Modal Masked Autoencoders for 2D Image and Video Understanding

Muhammad Abdullah Jamal, Omid MohareriWACV 2024

TL;DR

This paper introduces M³3D, a novel self-supervised pre-training framework that leverages both RGB and depth data to learn rich 3D priors for vision tasks. By combining masked image modeling with cross-modal contrastive learning, M³3D significantly improves performance on various downstream tasks like semantic segmentation and action recognition, especially in low-data regimes.

What Problem Does This Paper Address?

Modern computer vision models, particularly Vision Transformers (ViTs), have achieved remarkable success through self-supervised learning, often using masked autoencoders (MAE) to reconstruct missing parts of images or videos. However, these approaches typically rely on single-modality data. This overlooks the valuable geometric information present in RGB-D (color and depth) sensors, which can provide crucial 3D priors and reduce view-dependent biases. While some methods have explored RGB-D data, they often require camera pose registration or are limited to specific tasks. This paper aims to bridge this gap by developing a more general and effective self-supervised pre-training method that integrates 3D priors from RGB-D data into ViTs for a wider range of 2D image and video understanding tasks.

Methods

M³3D builds upon the Masked Autoencoder (MAE) paradigm but extends it to multi-modal RGB-D inputs. The core idea is to:

  • Multi-modal Masked Image Modeling (MIM): Randomly mask patches in both RGB and depth modalities. An encoder (either modality-specific or shared) processes the unmasked patches, and a decoder reconstructs the masked regions. This forces the model to learn robust representations that capture both visual and geometric information.
  • Cross-Modal Representation Learning: To further enhance the correspondence between RGB and depth, M³3D incorporates two strategies:
  • Contrastive Learning: Inspired by InfoNCE loss, this aims to align features from corresponding RGB and depth patches while pushing dissimilar patches apart. This helps create modality-invariant features.
  • Matching Loss: Similar to video-text matching, this loss predicts whether an RGB-depth pair is matched or not, acting as a binary classifier to reinforce cross-modal understanding.

The pre-training can be done in one or two stages. For video tasks, a single-stage approach is used. For image tasks, a two-stage approach is employed: first, contrastive learning for a few epochs, followed by masked image modeling with the encoder initialized from the first stage.

Datasets

The paper uses the following datasets:

  • Pre-training:
    • ScanNet: A large-scale dataset of RGB-D frames from indoor scenes, used for image-based pre-training.
  • UCF-101: A benchmark dataset for video action recognition, used for video-based pre-training.
  • OR-AR: A dataset for surgical video action detection, used for video-based pre-training.
  • Fine-tuning/Evaluation:
    • ScanNet: Used for 2D semantic segmentation.
  • NYUv2: Used for depth estimation and 2D semantic segmentation.
  • UCF-101: Used for video action recognition.
  • OR-AR: Used for surgical video action detection.

Key Insights and Ideas

  • 3D Priors are Crucial for 2D Tasks: The paper strongly suggests that incorporating explicit 3D geometric information, even from a single RGB-D view, significantly boosts performance on 2D tasks like semantic segmentation, going beyond what standard MAE can achieve.
  • Cross-Modal Learning is Key: Simply masking and reconstructing isn't enough. The contrastive and matching losses are vital for learning truly aligned and complementary representations between RGB and depth, leading to better transferability.
  • Data Efficiency: M³3D demonstrates superior data efficiency, achieving strong results even with limited labeled data during fine-tuning, which is a significant advantage in many real-world scenarios.
  • Generalizability: The framework is shown to be versatile, effectively transferring learned representations to diverse tasks (action recognition, detection, segmentation, depth estimation) and datasets, including out-of-domain ones.

Key Results

  • ScanNet Semantic Segmentation: M³3D achieved a mean Intersection over Union (mIoU) of 67.3%, outperforming Mask3D (+1.3 mIoU) and MAE (+2.8 mIoU).
  • NYUv2 Semantic Segmentation: M³3D achieved an mIoU of 51.2%, outperforming Mask3D and other state-of-the-art methods.
  • NYUv2 Depth Estimation: M³3D achieved an 81% accuracy (for pixels with error ratio below 1.25), outperforming CroCo (85.6% vs 86.7%) and Mask3D.
  • UCF-101 Video Action Recognition: M³3D achieved 96.3% top-1 accuracy, outperforming VideoMAE.
  • OR-AR Surgical Video Action Detection: M³3D achieved 80.90 mAP with only 5% labeled data, demonstrating significant data efficiency compared to VideoMAE.
  • Data Efficiency on ScanNet: M³3D recovered over 80% of the full labeled performance when fine-tuned with only 20% of the training data.

Limitations and Future Work

  • Depth Data Requirement: The primary limitation is the reliance on RGB-D data for pre-training. While the paper shows effectiveness, it's not directly applicable to RGB-only datasets without adaptation.
  • Future Work: The authors suggest extending the approach to applications with datasets that have more than two modalities, indicating a path towards even more comprehensive multi-modal learning. They also hypothesize that performance could be further enhanced with improved depth estimation models or readily available high-quality depth data.

Summary updated

BibTeX
@inproceedings{jamal2024m,
  author = {Muhammad Abdullah Jamal and Omid Mohareri},
  title = {{M33D: Learning 3D Priors Using Multi-Modal Masked Autoencoders for 2D Image and Video Understanding}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure