Letting 3D Guide the Way: 3D Guided 2D Few-Shot Image Classification

Jiajing Chen, Minmin Yang, Senem VelipasalarWACV 2024

TL;DR

This paper introduces a novel 3D-guided 2D few-shot image classification approach (3DG2D) that leverages 3D mesh data as support sets to improve classification accuracy on unseen 2D images. By generating multiple views from 3D models and incorporating an Angle Inference Network (AINet), the method effectively addresses the limitations of traditional 2D-only few-shot learning, achieving state-of-the-art results.

What Problem Does This Paper Address?

Traditional few-shot image classification (FSC) aims to classify new objects using only a few labeled examples (support set). However, existing methods typically rely on 2D images for both support and query sets, drawn from the same image pool. This approach suffers from two main issues:

  1. Viewpoint Mismatch: Support and query images can be captured from vastly different angles, making them appear dissimilar and hindering accurate matching.
  2. Dataset Overfitting: When support and query images come from the same pool, models can overfit to dataset-specific characteristics (e.g., color, texture, view angle), leading to poor generalization to real-world scenarios.

Humans, on the other hand, naturally leverage their 3D understanding of objects to recognize them from various viewpoints. This paper seeks to bridge this gap by incorporating 3D information into the FSC pipeline.

Methods

The core of the proposed method, 3D Guided 2D (3DG2D) Few-Shot Image Classification, involves using 3D mesh data as the support set instead of 2D images.

  • 3D Mesh Support Set: For each object in the support set, the authors generate 14 projection images from different viewpoints using a 3D mesh. This provides a richer representation of the object compared to a single 2D image.
  • Angle Inference Network (AINet): A significant challenge with using multiple projections is their inherent variance. To address this, AINet is introduced. It first infers the view angle of a query image and then assigns more attention to support projections that share a similar view angle. This is achieved through a novel angle inference loss ($L_A$) that complements the standard classification loss ($L_c$).
  • Pretraining: A ResNet backbone is pretrained using a strategy inspired by contrastive learning, focusing on learning robust feature representations from 2D projections of training class 3D meshes. This pretraining avoids exposing the model to the exact test classes, preventing misleadingly high performance.

The overall loss function is a weighted sum of the classification loss and the angle inference loss: $L = (1 - \alpha) * L_c + \alpha * L_a$. During testing, AINet uses the inferred angle probabilities to re-weight the class prototypes, leading to more accurate predictions.

Datasets

The proposed method was evaluated on the following datasets:

  • ModelNet40: A widely used dataset of 3D CAD models.
  • Toys4K: A dataset containing 3D models of toys.
  • ShapeNet: A large-scale dataset of 3D shapes with rich annotations.

The authors also conducted experiments using real-world RGB images collected from the web as query images, demonstrating the method's applicability beyond synthetic projections.

Key Insights and Ideas

  • 3D as a Superior Support Set: The paper convincingly demonstrates that using 3D mesh data, which inherently captures multi-view information, is a more robust and informative support set for few-shot learning than traditional 2D images.
  • Angle Matters: The analysis of angle and shape variety highlights that while both contribute to improved accuracy, angle variety can be more effectively achieved by leveraging multiple views from a single 3D model, rather than solely increasing the number of 2D shots.
  • The Power of Angle Inference: AINet's ability to infer query image angles and prioritize matching support projections from similar viewpoints is crucial for handling the variance introduced by multi-view projections, leading to significant performance gains.
  • Addressing Real-World Challenges: By using 3D meshes and generating projections with varying colors, textures, and viewpoints (distinct from query images), the approach better simulates real-world scenarios and mitigates dataset overfitting.

Key Results

The 3DG2D approach with AINet consistently outperformed state-of-the-art baselines across ModelNet40, Toys4K, and ShapeNet datasets. For instance, on ModelNet40 in the 1-shot setting, AINet achieved a mean accuracy of 65.34%, outperforming the second-best model by a notable margin. Similar improvements were observed in the 3-shot settings and on other datasets. The method also showed strong performance when tested with real-world RGB images as queries.

Limitations and Future Work

  • Computational Cost: Generating 14 projections per 3D mesh and the complexity of AINet might introduce higher computational overhead compared to purely 2D methods.
  • Reliance on 3D Models: The current approach relies on the availability of 3D models for the support set. While the paper shows promising results with web-scraped RGB images, further research could explore methods to infer 3D information from 2D images more effectively for FSC.
  • Future Work: The authors suggest extending experiments to RGB images with more classes and exploring more advanced techniques for integrating 3D information into few-shot learning.

Summary updated

BibTeX
@inproceedings{chen2024letting,
  author = {Jiajing Chen and Minmin Yang and Senem Velipasalar},
  title = {{Letting 3D Guide the Way: 3D Guided 2D Few-Shot Image Classification}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure