TL;DR
This paper tackles the challenge of efficiently transferring pre-trained RGB Vision Transformers (ViTs) to RGB-D 3D object recognition tasks. They demonstrate that a "late fusion" approach, where RGB and depth information are processed separately by the ViT encoder and then combined, significantly outperforms "early fusion" and achieves new state-of-the-art results on the Washington RGB-D Objects (ROD) dataset.
What Problem Does This Paper Address?
Vision Transformers (ViTs) have shown remarkable success in computer vision, but their application to RGB-D (color and depth) 3D object recognition is less explored. A major hurdle is the lack of large-scale RGB-D pre-training datasets, forcing researchers to rely on transferring knowledge from RGB-only pre-trained models. Existing methods often use computationally intensive multi-task pre-training or early fusion strategies, which combine RGB and depth information before feeding it into the ViT encoder. This paper questions whether this early fusion is optimal, especially when transferring from RGB-only pre-trained models to RGB-D tasks with limited data, and explores more efficient fusion strategies.
Methods
The core of this work is investigating how to best fuse RGB and depth information within a ViT architecture when transferring from an RGB-only pre-trained model. They propose two main fusion strategies:
- Early Fusion: RGB and depth are processed by separate patch embedding layers and then fused before entering the Transformer encoder. This can be done with either a "dual-embedder" (separate projection for each modality) or a "joint-embedder" (fusing channels before a single projection).
- Late Fusion: RGB and depth are processed separately by the Transformer encoder, and their final
<CLS>token representations are fused after the encoder, before being fed to the classification head. Various fusion operations like averaging, max pooling, or concatenation are explored for this late-stage combination.
The authors also experiment with different depth representations, including raw depth, HHA transformations, and surface normals, finding that surface normals generally perform best.
Datasets
The primary dataset used for training and evaluation is the Washington RGB-D Objects (ROD) dataset. This dataset contains approximately 41,877 views of 300 object instances across 51 categories.
Additional datasets used for evaluation include:
- ARID40k: A subset of ARID, containing 40,713 RGB-D crop pairs, designed for robotic-specific applications.
- Homebreweddb (HB): A dataset featuring 3D-reconstructed models, used here for RGB-D object recognition with surface normal depth.
- OCID: A dataset with 96 cluttered scenes, containing RGB-D crops from ARID and the YCB dataset, posing challenges due to occlusions.
- SynROD: A synthetic version of the ROD dataset, used for synthetic-to-real domain adaptation experiments.
Key Insights and Ideas
- Late Fusion Dominance: The most significant finding is that late fusion consistently outperforms early fusion for RGB → RGB-D transfer, especially in low-data regimes. This suggests that modifying the core encoder's understanding of RGB input (as early fusion does) is detrimental when transferring from RGB-only pre-training. Late fusion allows the encoder to leverage its RGB pre-training more effectively.
- Depth Representation Matters: Surface normal reconstructions of depth data generally yield better results than raw depth or HHA transformations, indicating that geometric details are crucial for effective fusion.
- ViT's Adaptability: The paper highlights the inherent flexibility of ViTs to learn cross-modal interactions at the downstream task level, rather than requiring complex pre-training strategies.
- Lifelong Learning Potential: The late fusion approach shows promise for open-ended lifelong learning scenarios, where the model can incrementally learn new objects without significant forgetting.
- Synthetic-to-Real Transfer: Late fusion also proves effective in synthetic-to-real domain adaptation, enabling models trained on synthetic data to generalize better to real-world data with minimal fine-tuning.
Key Results
- State-of-the-Art on ROD: The proposed late fusion approach with ViT-B achieves a new state-of-the-art top-1 accuracy of 95.4% on the Washington RGB-D Objects (ROD) benchmark.
- Significant Improvement over Early Fusion: Late fusion consistently outperforms early fusion, with the gap widening as the amount of training data decreases. For example, on OCID, late fusion is 8.6% better than early fusion.
- Lifelong Learning Performance: The late fusion model significantly outperforms previous works in lifelong learning scenarios on ROD, achieving a delta of over 8% in accuracy metrics.
- Synthetic-to-Real Adaptation: With just 10-20 labeled examples per object instance, the late fusion model trained on synthetic data can achieve performance comparable to or exceeding models trained on significantly more real data.
Limitations and Future Work
- Computational Cost of Late Fusion: While effective, late fusion requires running the ViT encoder twice (once for RGB, once for depth), which increases computational cost compared to a single forward pass.
- Fusion Operation Choice: The paper notes that concatenation in late fusion outperforms other methods but doubles the classifier's hidden size.
- Future Directions: The authors suggest exploring more sophisticated fusion techniques beyond simple pooling (e.g., hierarchical or attention-based fusion), investigating parameter-efficient fine-tuning methods (like adapters), and extending this work to domains outside the ImageNet class distribution. They also aim to improve computational efficiency and generalization to novel domains.
Summary updated