Doodle to Object: Practical Zero-Shot Sketch-Based 3D Shape Retrieval

Bingrui Wang, Yuan ZhouAAAI 2023 · DOI

TL;DR

This paper tackles the challenging problem of Zero-Shot Sketch-Based 3D Shape Retrieval (ZS-SBSR), where users sketch an object and the system retrieves corresponding 3D models from categories it has never seen before. They introduce a new dataset, Doodle2Object (D2O), and a novel Prototype Contrastive Learning (PCL) method that effectively bridges the semantic gap between abstract sketches and detailed 3D shapes.

What Problem Does This Paper Address?

Sketch-Based 3D Shape Retrieval (SBSR) aims to find 3D models based on user-drawn sketches. This is inherently difficult due to the significant domain gap between 2D sketches and 3D shapes, and the abstraction level of sketches. Existing SBSR datasets are often small and lack variety, making it hard to train robust models. Furthermore, traditional SBSR methods struggle with the "Zero-Shot" (ZS) setting, where the system must retrieve objects from categories it hasn't encountered during training. This ZS requirement is crucial for practical applications where users might sketch objects outside the system's known categories. The paper addresses this by proposing a method and dataset that can handle ZS-SBSR, aiming to improve retrieval accuracy and address data scarcity.

Methods

The core of the paper is the Prototype Contrastive Learning (PCL) method. PCL operates in two main stages:

  1. Intra-Domain Feature Extraction: This stage uses separate feature extractors for sketches and 3D shapes. The sketch extractor, "Sketch-a-net," is a CNN-RNN architecture designed to capture both spatial and temporal sketch information. The 3D shape extractor uses a multi-view approach with a ResNet-50 backbone and dual attention mechanisms. The PCL method then trains these extractors by pulling samples of the same class closer to their respective class prototypes while pushing them away from prototypes of different classes. This is achieved through a contrastive loss function that leverages memory banks of features and dynamically updated prototypes.
  2. Cross-Domain Feature Alignment: After training the individual domain extractors, this stage aligns the feature spaces of sketches and 3D shapes. It uses the PCL concept again, but this time to map features from both domains into a common semantic space. Sample pairs (sketch-3D shape) from common classes are used to train mapping networks. The goal is to ensure that semantically similar sketches and 3D shapes are located close to each other in this shared space, even for unseen classes.

Datasets

  • SHREC'13 and SHREC'14: These are existing, widely used benchmark datasets for SBSR. The paper notes their limitations, including data scarcity in some classes and the lack of temporal information in sketches.
  • Doodle2Object (D2O): This is a newly introduced dataset by the authors. It comprises 8,992 3D shapes and over 7 million sketches spanning 50 categories. D2O is augmented using ModelNet40 for 3D shapes and QuickDraw for sketches, aiming to provide more realistic sketches with temporal information and larger sample sizes per class, thus alleviating data scarcity issues and better mimicking real-world scenarios.

Key Insights and Ideas

  • Bridging the Semantic Gap with Prototypes: The paper highlights that using class prototypes as anchors for contrastive learning is effective in pulling together semantically similar items across domains, even when those domains are vastly different (sketches vs. 3D shapes).
  • The Importance of Cross-Domain Alignment: Explicitly aligning the feature spaces of sketches and 3D shapes is crucial for effective retrieval, especially in the ZS setting. Simply learning good features within each domain isn't enough.
  • Temporal Sketch Information Matters: While many existing SBSR methods treat sketches as static images, the paper demonstrates that incorporating temporal stroke information (using RNNs) can improve retrieval performance, particularly for more abstract sketches like those from QuickDraw.
  • D2O Dataset's Value: The creation of the D2O dataset is a significant contribution, providing a more realistic and larger-scale benchmark for ZS-SBSR research, addressing the limitations of previous datasets.

Key Results

The proposed PCL method, when applied to ZS-SBSR, consistently outperforms state-of-the-art methods across the SHREC'13, SHREC'14, and the new D2O datasets. For instance, on the D2O dataset, the PCL method achieved significantly higher scores in metrics like Nearest Neighbor (NN) and Mean Average Precision (mAP) compared to other methods. The paper also shows that incorporating temporal sketch information (using RNNs) leads to improved retrieval performance, especially when dealing with abstract sketches.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while D2O provides more realistic sketches, they are still derived from existing datasets (QuickDraw). Real-world sketches from users might exhibit even greater variability.
  • The temporal sketch information experiments were conducted on D2O, which contains temporal data. For SHREC'13/14, which lack this, the temporal aspect was not utilized.
  • Future Work:
    • The authors plan to conduct more experiments with data collected from actual user environments.
  • They suggest exploring AutoML feature extractors to further ease the customization of networks for specific modalities.

Summary updated

BibTeX
@inproceedings{wang2023doodle,
  author = {Bingrui Wang and Yuan Zhou},
  title = {{Doodle to Object: Practical Zero-Shot Sketch-Based 3D Shape Retrieval}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25344},
}
Figure