Abstract
In this work, we tackle the problem of single image-based 3D shape retrieval (IBSR), where we seek to find the most matched shape of a given single 2D image from a shape repository. Most existing works learn to embed 2D images and 3D shapes into a common feature space and perform metric learning using a triplet loss. Inspired by the success of contrastive learning in self-supervised representation learning, we propose a novel IBSR pipeline leveraging cross-modal contrastive learning. We address the challenges of domain gap and feature entanglement by introducing a strong "color transfer" data augmentation and a novel category-level contrastive loss. Our approach achieves state-of-the-art performance on three popular benchmarks—Pix3D, Stanford Cars, and Comp Cars—outperforming previous methods by 4% to 15% in retrieval accuracy.
Introduction: The Challenge of IBSR
Image-Based 3D Shape Retrieval (IBSR) aims to retrieve the exact 3D CAD model matching an object in a single 2D photo. This task is complicated by the "domain gap": real-world photos contain lighting, textures, and backgrounds absent in synthetic 3D models. Traditional methods often rely on triplet loss, which is sensitive to "hard negative mining" and can lead models to match objects based on superficial features like color rather than underlying geometry.
Methodology: Contrastive Learning and Color Decoupling
The proposed framework replaces standard triplet loss with a Cross-Modal Instance and Category Contrastive Learning pipeline.
Key Components:
- Color Transfer Augmentation: To force the model to focus on geometry, the authors use a technique that transfers the color palette of a random source image onto the query image. This provides the strong augmentation required for contrastive learning and effectively decorrelates appearance from shape.
- Multi-View Grayscale Representation: 3D shapes are represented as a set of 12 multi-view grayscale rendered images, simplifying the cross-modal matching task.
- Query-Specific Attention: An attention module merges multi-view 3D features based on the specific 2D query image, creating a dynamic embedding.
- Dual-Level Contrastive Loss:
- Instance Loss: Pulls a query image toward its matching 3D shape while pushing it away from all other shapes in the batch.
- Category Loss: A supervised contrastive loss that pulls the query toward all shapes of the same category (e.g., all chairs), helping the model learn broader structural features and distinguish between similar-looking objects from different classes.
Experimental Results
The model was evaluated on three challenging real-world datasets: Pix3D (furniture), Stanford Cars, and Comp Cars (fine-grained car models).
As shown in the results table, the proposed method significantly outperforms previous leaders like HEG-TS and LFD. On the Pix3D (Bed) category, Top-1 accuracy improved from 65.3% to 73.3%. The gains were even more pronounced in fine-grained tasks; on Stanford Cars, Top-1 accuracy reached 83.4%, a massive leap from the previous state-of-the-art of 68.4%. Furthermore, the model demonstrated strong generalization on unseen 3D models from ShapeNet, indicating it learns robust geometric features rather than just memorizing training samples.
Conclusion and Future Work
By leveraging dual-level contrastive learning and color-decoupling augmentations, this work sets a new benchmark for 3D shape retrieval. While the method currently depends on accurate object masks, future directions include explicitly integrating 3D pose estimation and optimizing the framework for faster inference using just-in-time compiled deep learning frameworks like Jittor.
Summary updated