Democratising 2D Sketch to 3D Shape Retrieval Through Pivoting

Pinaki Nath Chowdhury, Ayan Kumar Bhunia, Aneeshan Sain, Subhadeep Koley, Tao Xiang, Yi-Zhe Song

Introduction

Retrieving a specific 3D model from a hand-drawn sketch is a cornerstone task for applications in augmented reality, computer-aided design, and digital entertainment. However, the field has long been bottlenecked by two primary issues: the extreme scarcity of fine-grained paired data (sketches matched to specific 3D models) and the inherent variance in user sketching skill and perspective.

Traditional supervised methods require massive datasets of sketches paired with 3D models from specific viewpoints. Collecting such data is prohibitively expensive and often fails to generalize to the "messy" nature of amateur drawings or arbitrary camera angles. This paper proposes a "democratic" approach that bypasses the need for manual paired supervision by leveraging existing, unrelated datasets through two core mechanisms: Pivoting and Lifting.

Overview of the pivoting strategy to bridge the gap between sketches and 3D shapes using 2D photos.

Methodology

The proposed framework eliminates the requirement for direct (Sketch, 3D Shape) pairs by utilizing 2D images as an intermediate domain.

Pivoting Across Domains

The "Pivoting" strategy bridges two distinct data sources:

  1. Domain A: Existing datasets of paired 2D sketches and photos.
  2. Domain B: 3D shape datasets (e.g., ShapeNet) and their rendered 2D projections.

By employing a shared 2D encoder across sketches, photos, and projections, the model learns a unified feature space. A KL-divergence loss is applied to align the distribution of distances in the sketch-photo domain with those in the projection-shape domain. This effectively allows the system to "pivot" through the 2D image domain to perform 3D retrieval.

Lifting to 3D Space

To ensure the 2D encoder understands the underlying geometry of the objects—making it robust to viewpoint changes—the authors introduce a "Lifting" module. This module utilizes an auxiliary task: Blind Perspective-n-Points (BPnP).

The proposed architecture featuring Pivoting and Lifting modules for 3D-aware sketch representation.

During training, the model predicts 3D coordinates and confidence weights for sampled 2D points on an image projection. A BPnP solver then estimates the object's pose. The error between the predicted and ground-truth pose is backpropagated to the 2D encoder. This geometric regularization forces the encoder to learn spatial relationships, ensuring that a sketch of a chair from the side is mapped near a 3D model of that same chair, regardless of the sketch's orientation.

Experimental Results

The model was evaluated on several benchmarks, including QMUL-Chair-V2 and ShapeNet, with fine-grained testing on the FG-SBSR dataset.

  • Zero-Shot Performance: In a zero-shot setting—where the model never saw a single sketch-3D pair during training—it achieved an Acc@1 of 55.79% on the 'Chairs' category. This performance is remarkably close to the fully supervised "all-shot" baseline of 58.53%, demonstrating the effectiveness of the pivoting strategy.
  • Geometric Awareness: The "Lifting" technique provided a significant performance boost over standard 2D encoders. This confirms that injecting "3D common sense" into the feature extraction process is vital for handling the viewpoint variance inherent in free-hand sketching.
  • Generalization: On the SHREC’13 category-level retrieval task, the model achieved a Nearest Neighbor (NN) score of 55.3%, proving its utility beyond fine-grained instance retrieval.

Discussion and Limitations

While the results are promising, challenges remain in cross-category generalization. For instance, performance on 'Lamps' (21.63% Acc@1) was lower because the pivot dataset primarily contained chairs. The model's learned "sketch-to-photo" mapping did not transfer perfectly to significantly different geometries.

Future work may involve integrating large-scale foundation models like CLIP to provide a more universal pivot, potentially enabling the system to work across any object category without requiring category-specific paired photo-sketch data.

Conclusion

By reframing 2D-to-3D retrieval as a cross-domain alignment problem mediated by 2D images and geometric lifting, this work provides a scalable path forward. It moves the field away from expensive data collection and toward a more flexible, "democratic" system capable of understanding amateur sketches in a 3D context.

Summary updated

Abstract

This paper studies the problem of 2D sketch to 3D shape retrieval, but with a focus on democratising the process. We would like this democratisation to happen on two fronts: (i) to remove the need for large-scale specifically sourced 2D sketch and 3D shape datasets, and (ii) to remove restrictions on how well the user needs to sketch and from what viewpoints. The end result is a system that is trainable using existing datasets, and once trained allows users to sketch regardless of drawing skills and without restriction on view angle. We achieve all this via a clever use of pivoting, along with novel designs that injects 3D understanding of 2D sketches into the system. We perform pivoting using two existing datasets, each from a distant research domain to the other: 2D sketch and photo pairs from the sketch-based image retrieval field (SBIR), and 3D shapes from ShapeNet. It follows that the actual feature pivoting happens on photos from the former and 2D projections from the latter. Doing this already achieves most of our democratisation challenge – the level of 2D sketch abstraction embedded in SBIR dataset offers demoralization on drawing quality, and the whole thing works without a specifically sourced 2D sketch and 3D model pair. To further achieve democratisation on sketching viewpoint, we “lift” 2D sketches to 3D space using Blind Perspective-n-Points (BPnP) that injects 3D-aware information into the sketch encoder. Results show ours achieves competitive performance compared with fully-supervised baselines, while meeting all set democratisation goals.
BibTeX
@inproceedings{chowdhury2023democratising,
  author = {Pinaki Nath Chowdhury and Ayan Kumar Bhunia and Aneeshan Sain and Subhadeep Koley and Tao Xiang and Yi-Zhe Song},
  title = {{Democratising 2D Sketch to 3D Shape Retrieval Through Pivoting}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.02127},
}
Figure