Enhancing Fine-Grained 3D Object Recognition Using Hybrid Multi-Modal Vision Transformer-CNN Models

Hamidreza Kasaei, Songsong Xiong, Georgios Tziafas, Hamidreza Kasaei 0001

TL;DR

This paper tackles the challenging problem of fine-grained 3D object recognition for robots, which is crucial for tasks in human-centric environments. The authors introduce a novel hybrid approach combining Vision Transformers (ViT) and Convolutional Neural Networks (CNNs) to leverage both global and local features from RGB-D data. They also address the scarcity of suitable datasets by creating two new synthetic RGB-D datasets for restaurant items and shoes, demonstrating superior performance over unimodal baselines.

What Problem Does This Paper Address?

Robots operating in everyday environments, like stores or homes, need to recognize objects with high precision. This is especially true for "fine-grained" recognition, where objects within the same basic category (e.g., different types of shoes or restaurant cutlery) have subtle visual differences. This task is inherently difficult due to high intra-category similarity (objects within a class look alike) and low inter-category dissimilarity (different classes look similar). Furthermore, existing 3D object recognition datasets are often limited in scale or scope, hindering the development of robust models. This paper aims to improve fine-grained 3D object recognition by proposing a more powerful model architecture and by contributing new, larger datasets.

Methods

The core of the proposed method is a hybrid multi-modal approach that fuses information from both Vision Transformers (ViT) and Convolutional Neural Networks (CNNs). The intuition is that ViTs excel at capturing global context and long-range dependencies, while CNNs are adept at extracting local features and textures.

Here's how it works:

  1. Data Input: The model takes RGB-D images as input.
  2. Feature Extraction:
    • One branch uses a ViT (specifically, the MAE model) to encode the RGB image, capturing global object features.
  3. Another branch uses a CNN (specifically, DenseNet) to encode either the RGB image or the depth image, focusing on local details.
  4. Fusion: The feature representations from the ViT and CNN branches are fused. The paper explores both element-wise pooling (average or maximum) and concatenation to create a comprehensive multi-modal representation.
  5. Classification: A k-Nearest Neighbors (kNN) classifier with the Motyka distance function and k=1 was found to be the most effective for classification.

The authors also explored two main configurations:

  • Approach I: ViT for RGB + CNN for Depth.
  • Approach II: ViT for RGB + CNN for RGB-D (where RGB and Depth are processed together by the CNN).

Datasets

The paper addresses the lack of suitable datasets by introducing two new synthetic RGB-D datasets:

  • Restaurant Objects Dataset: Contains 1200 instance views across 20 categories of restaurant items.
  • Shoe Objects Dataset: Contains 7200 instance views across 120 categories of shoes.

These datasets were generated using a simulation environment (Gazebo) with a robot arm and a Kinect camera, capturing partial views of objects moved along a trajectory. The authors emphasize that these objects have subtle differences, making them challenging for fine-grained recognition.

Key Insights and Ideas

  • Synergy of ViT and CNN: Combining ViTs and CNNs is more effective than using either alone for fine-grained 3D object recognition. This highlights the complementary strengths of these architectures in capturing both global context and local details.
  • RGB-D is Crucial for Fine-Grained Tasks: The use of depth information alongside RGB significantly boosts recognition accuracy, especially for objects with subtle visual differences.
  • Dataset Creation is Key: The scarcity of large-scale, fine-grained RGB-D datasets is a major bottleneck. The authors' contribution of new datasets is vital for advancing research in this area.
  • kNN with Motyka Distance is Effective: For the chosen feature representations, a simple kNN classifier with a specific distance metric proved to be a strong performer, simplifying the downstream classification task.

Key Results

The proposed hybrid multi-modal approach consistently outperformed unimodal CNN-only and ViT-only baselines on both the restaurant and shoe datasets.

  • Restaurant Dataset: Achieved a recognition accuracy of 94.50% with the hybrid approach.
  • Shoe Dataset: Achieved a recognition accuracy of 93.51% with the hybrid approach.

The paper also demonstrated the practical utility of their method by integrating it into a dual-arm robot framework for both simulated and real-world tasks, including sorting shoes and robot-assisted packaging, showcasing successful fine-grained object manipulation.

Limitations and Future Work

  • Computation Time: While the hybrid approach offers superior accuracy, it is noted that multi-modal approaches are generally slower than single-modality approaches. The paper mentions that element-wise max pooling is faster than average pooling.
  • Future Work: The authors suggest investigating the possibility of improving performance by considering different views of the object from various viewpoints, implying a potential for incorporating more sophisticated multi-view fusion techniques.

Summary updated

Abstract

Robots operating in human-centered environments, such as retail stores, restaurants, and households, are often required to distinguish between similar objects in different contexts with a high degree of accuracy. However, fine-grained object recognition remains a challenge in robotics due to the high intra-category and low inter-category dissimilarities. In addition, the limited number of fine-grained 3D datasets poses a significant problem in addressing this issue effectively. In this paper, we propose a hybrid multi-modal Vision Transformer (ViT) and Convolutional Neural Networks (CNN) approach to improve the performance of fine-grained visual classification (FGVC). To address the shortage of FGVC 3D datasets, we generated two synthetic datasets. The first dataset consists of 20 categories related to restaurants with a total of 100 instances, while the second dataset contains 120 shoe instances. Our approach was evaluated on both datasets, and the results indicate that it outperforms both CNN-only and ViT-only baselines, achieving a recognition accuracy of 94.50 % and 93.51 % on the restaurant and shoe datasets, respectively. Additionally, we have made our FGVC RGB-D datasets available to the research community to enable further experimentation and advancement. Furthermore, we successfully integrated our proposed method with a robot framework and demonstrated its potential as a fine-grained perception tool in both simulated and real-world robotic scenarios.
BibTeX
@inproceedings{kasaei2023enhancing,
  author = {Hamidreza Kasaei and Songsong Xiong and Georgios Tziafas and Hamidreza Kasaei},
  title = {{Enhancing Fine-Grained 3D Object Recognition Using Hybrid Multi-Modal Vision Transformer-CNN Models}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10342235},
}
Figure