Single Image 3D Shape Retrieval via Cross-Modal Instance and Category Contrastive Learning

Ming-Xian Lin, Jie Yang, He Wang, Yu-Kun Lai, Rongfei Jia, Binqiang Zhao, Lin GaoICCV 2021 · OpenReview

Abstract

In this work, we tackle the problem of single image-based 3D shape retrieval (IBSR), where we seek to find the most matched shape of a given single 2D image from a shape repository. Most existing works learn to embed 2D images and 3D shapes into a common feature space and perform metric learning using a triplet loss. Inspired by the success of contrastive learning in self-supervised representation learning, we propose a novel IBSR pipeline leveraging cross-modal contrastive learning. We address the challenges of domain gap and feature entanglement by introducing a strong "color transfer" data augmentation and a novel category-level contrastive loss. Our approach achieves state-of-the-art performance on three popular benchmarks—Pix3D, Stanford Cars, and Comp Cars—outperforming previous methods by 4% to 15% in retrieval accuracy.

Introduction: The Challenge of IBSR

Image-Based 3D Shape Retrieval (IBSR) aims to retrieve the exact 3D CAD model matching an object in a single 2D photo. This task is complicated by the "domain gap": real-world photos contain lighting, textures, and backgrounds absent in synthetic 3D models. Traditional methods often rely on triplet loss, which is sensitive to "hard negative mining" and can lead models to match objects based on superficial features like color rather than underlying geometry.

Methodology: Contrastive Learning and Color Decoupling

The proposed framework replaces standard triplet loss with a Cross-Modal Instance and Category Contrastive Learning pipeline.

The proposed IBSR pipeline featuring color transfer augmentation and dual-level contrastive loss.

Key Components:

  • Color Transfer Augmentation: To force the model to focus on geometry, the authors use a technique that transfers the color palette of a random source image onto the query image. This provides the strong augmentation required for contrastive learning and effectively decorrelates appearance from shape.
  • Multi-View Grayscale Representation: 3D shapes are represented as a set of 12 multi-view grayscale rendered images, simplifying the cross-modal matching task.
  • Query-Specific Attention: An attention module merges multi-view 3D features based on the specific 2D query image, creating a dynamic embedding.
  • Dual-Level Contrastive Loss:
    • Instance Loss: Pulls a query image toward its matching 3D shape while pushing it away from all other shapes in the batch.
  • Category Loss: A supervised contrastive loss that pulls the query toward all shapes of the same category (e.g., all chairs), helping the model learn broader structural features and distinguish between similar-looking objects from different classes.

Experimental Results

The model was evaluated on three challenging real-world datasets: Pix3D (furniture), Stanford Cars, and Comp Cars (fine-grained car models).

Quantitative comparison of the proposed method against state-of-the-art techniques on Pix3D, Comp Cars, and Stanford Cars datasets.

As shown in the results table, the proposed method significantly outperforms previous leaders like HEG-TS and LFD. On the Pix3D (Bed) category, Top-1 accuracy improved from 65.3% to 73.3%. The gains were even more pronounced in fine-grained tasks; on Stanford Cars, Top-1 accuracy reached 83.4%, a massive leap from the previous state-of-the-art of 68.4%. Furthermore, the model demonstrated strong generalization on unseen 3D models from ShapeNet, indicating it learns robust geometric features rather than just memorizing training samples.

Conclusion and Future Work

By leveraging dual-level contrastive learning and color-decoupling augmentations, this work sets a new benchmark for 3D shape retrieval. While the method currently depends on accurate object masks, future directions include explicitly integrating 3D pose estimation and optimizing the framework for faster inference using just-in-time compiled deep learning frameworks like Jittor.

Summary updated

Abstract

In this work, we tackle the problem of single image-based 3D shape retrieval (IBSR), where we seek to find the most matched shape of a given single 2D image from a shape repository. Most of the existing works learn to embed 2D images and 3D shapes into a common feature space and perform metric learning using a triplet loss. Inspired by the great success in recent contrastive learning works on self-supervised representation learning, we propose a novel IBSR pipeline leveraging contrastive learning. We note that adopting such cross-modal contrastive learning between 2D images and 3D shapes into IBSR tasks is non-trivial and challenging: contrastive learning requires very strong data augmentation in constructed positive pairs to learn the feature invariance, whereas traditional metric learning works do not have this requirement. Moreover, object shape and appearance are entangled in 2D query images, thus making the learning task more difficult than contrasting single-modal data. To mitigate the challenges, we propose to use multi-view grayscale rendered images from the 3D shapes as a shape representation. We then introduce a strong data augmentation technique based on color transfer, which can significantly but naturally change the appearance of the query image, effectively satisfying the need for contrastive learning. Finally, we propose to incorporate a novel category-level contrastive loss that helps distinguish similar objects from different categories, in addition to classic instance-level contrastive loss. Our experiments demonstrate that our approach achieves the best performance on all the three popular IBSR benchmarks, including Pix3D, Stanford Cars, and Comp Cars, outperforming the previous state-of-the-art from 4% - 15% on retrieval accuracy.
BibTeX
@inproceedings{lin2021single,
  author = {Ming-Xian Lin and Jie Yang and He Wang and Yu-Kun Lai and Rongfei Jia and Binqiang Zhao and Lin Gao},
  title = {{Single Image 3D Shape Retrieval via Cross-Modal Instance and Category Contrastive Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=Kt1XUaqrQZ},
}
Figure