Early or Late Fusion Matters: Efficient RGB-D Fusion in Vision Transformers for 3D Object Recognition

Hamidreza Kasaei, Georgios Tziafas, Hamidreza Kasaei 0001

TL;DR

This paper tackles the challenge of efficiently transferring pre-trained RGB Vision Transformers (ViTs) to RGB-D 3D object recognition tasks. They demonstrate that a "late fusion" approach, where RGB and depth information are processed separately by the ViT encoder and then combined, significantly outperforms "early fusion" and achieves new state-of-the-art results on the Washington RGB-D Objects (ROD) dataset.

What Problem Does This Paper Address?

Vision Transformers (ViTs) have shown remarkable success in computer vision, but their application to RGB-D (color and depth) 3D object recognition is less explored. A major hurdle is the lack of large-scale RGB-D pre-training datasets, forcing researchers to rely on transferring knowledge from RGB-only pre-trained models. Existing methods often use computationally intensive multi-task pre-training or early fusion strategies, which combine RGB and depth information before feeding it into the ViT encoder. This paper questions whether this early fusion is optimal, especially when transferring from RGB-only pre-trained models to RGB-D tasks with limited data, and explores more efficient fusion strategies.

Methods

The core of this work is investigating how to best fuse RGB and depth information within a ViT architecture when transferring from an RGB-only pre-trained model. They propose two main fusion strategies:

  • Early Fusion: RGB and depth are processed by separate patch embedding layers and then fused before entering the Transformer encoder. This can be done with either a "dual-embedder" (separate projection for each modality) or a "joint-embedder" (fusing channels before a single projection).
  • Late Fusion: RGB and depth are processed separately by the Transformer encoder, and their final <CLS> token representations are fused after the encoder, before being fed to the classification head. Various fusion operations like averaging, max pooling, or concatenation are explored for this late-stage combination.

The authors also experiment with different depth representations, including raw depth, HHA transformations, and surface normals, finding that surface normals generally perform best.

Datasets

The primary dataset used for training and evaluation is the Washington RGB-D Objects (ROD) dataset. This dataset contains approximately 41,877 views of 300 object instances across 51 categories.

Additional datasets used for evaluation include:

  • ARID40k: A subset of ARID, containing 40,713 RGB-D crop pairs, designed for robotic-specific applications.
  • Homebreweddb (HB): A dataset featuring 3D-reconstructed models, used here for RGB-D object recognition with surface normal depth.
  • OCID: A dataset with 96 cluttered scenes, containing RGB-D crops from ARID and the YCB dataset, posing challenges due to occlusions.
  • SynROD: A synthetic version of the ROD dataset, used for synthetic-to-real domain adaptation experiments.

Key Insights and Ideas

  • Late Fusion Dominance: The most significant finding is that late fusion consistently outperforms early fusion for RGB → RGB-D transfer, especially in low-data regimes. This suggests that modifying the core encoder's understanding of RGB input (as early fusion does) is detrimental when transferring from RGB-only pre-training. Late fusion allows the encoder to leverage its RGB pre-training more effectively.
  • Depth Representation Matters: Surface normal reconstructions of depth data generally yield better results than raw depth or HHA transformations, indicating that geometric details are crucial for effective fusion.
  • ViT's Adaptability: The paper highlights the inherent flexibility of ViTs to learn cross-modal interactions at the downstream task level, rather than requiring complex pre-training strategies.
  • Lifelong Learning Potential: The late fusion approach shows promise for open-ended lifelong learning scenarios, where the model can incrementally learn new objects without significant forgetting.
  • Synthetic-to-Real Transfer: Late fusion also proves effective in synthetic-to-real domain adaptation, enabling models trained on synthetic data to generalize better to real-world data with minimal fine-tuning.

Key Results

  • State-of-the-Art on ROD: The proposed late fusion approach with ViT-B achieves a new state-of-the-art top-1 accuracy of 95.4% on the Washington RGB-D Objects (ROD) benchmark.
  • Significant Improvement over Early Fusion: Late fusion consistently outperforms early fusion, with the gap widening as the amount of training data decreases. For example, on OCID, late fusion is 8.6% better than early fusion.
  • Lifelong Learning Performance: The late fusion model significantly outperforms previous works in lifelong learning scenarios on ROD, achieving a delta of over 8% in accuracy metrics.
  • Synthetic-to-Real Adaptation: With just 10-20 labeled examples per object instance, the late fusion model trained on synthetic data can achieve performance comparable to or exceeding models trained on significantly more real data.

Limitations and Future Work

  • Computational Cost of Late Fusion: While effective, late fusion requires running the ViT encoder twice (once for RGB, once for depth), which increases computational cost compared to a single forward pass.
  • Fusion Operation Choice: The paper notes that concatenation in late fusion outperforms other methods but doubles the classifier's hidden size.
  • Future Directions: The authors suggest exploring more sophisticated fusion techniques beyond simple pooling (e.g., hierarchical or attention-based fusion), investigating parameter-efficient fine-tuning methods (like adapters), and extending this work to domains outside the ImageNet class distribution. They also aim to improve computational efficiency and generalization to novel domains.

Summary updated

Abstract

The Vision Transformer (ViT) architecture has established its place in computer vision literature, however, training ViTs for RGB-D object recognition remains an understudied topic, viewed in recent literature only through the lens of multi-task pretraining in multiple vision modalities. Such approaches are often computationally intensive, relying on the scale of multiple pretraining datasets to align RGB with 3D information. In this work, we propose a simple yet strong recipe for transferring pretrained ViTs in RGB-D domains for 3D object recognition, focusing on fusing RGB and depth representations encoded jointly by the ViT. Compared to previous works in multimodal Transformers, the key challenge here is to use the attested flexibility of ViTs to capture cross-modal interactions at the downstream and not the pretraining stage. We explore which depth representation is better in terms of resulting accuracy and compare early and late fusion techniques for aligning the RGB and depth modalities within the ViT architecture. Experimental results in the Washington RGB-D Objects dataset (ROD) demonstrate that in such RGB -> RGB-D scenarios, late fusion techniques work better than most popularly employed early fusion. With our transfer baseline, fusion ViTs score up to 95.4% top-1 accuracy in ROD, achieving new state-of-the-art results in this benchmark. We further show the benefits of using our multimodal fusion baseline over unimodal feature extractors in a synthetic-to-real visual adaptation as well as in an open-ended lifelong learning scenario in the ROD benchmark, where our model outperforms previous works by a margin of >8%. Finally, we integrate our method with a robot framework and demonstrate how it can serve as a perception utility in an interactive robot learning scenario, both in simulation and with a real robot.
BibTeX
@inproceedings{kasaei2023early,
  author = {Hamidreza Kasaei and Georgios Tziafas and Hamidreza Kasaei},
  title = {{Early or Late Fusion Matters: Efficient RGB-D Fusion in Vision Transformers for 3D Object Recognition}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10341422},
}
Figure