Introduction
Retrieving a specific 3D model from a hand-drawn sketch is a cornerstone task for applications in augmented reality, computer-aided design, and digital entertainment. However, the field has long been bottlenecked by two primary issues: the extreme scarcity of fine-grained paired data (sketches matched to specific 3D models) and the inherent variance in user sketching skill and perspective.
Traditional supervised methods require massive datasets of sketches paired with 3D models from specific viewpoints. Collecting such data is prohibitively expensive and often fails to generalize to the "messy" nature of amateur drawings or arbitrary camera angles. This paper proposes a "democratic" approach that bypasses the need for manual paired supervision by leveraging existing, unrelated datasets through two core mechanisms: Pivoting and Lifting.
Methodology
The proposed framework eliminates the requirement for direct (Sketch, 3D Shape) pairs by utilizing 2D images as an intermediate domain.
Pivoting Across Domains
The "Pivoting" strategy bridges two distinct data sources:
- Domain A: Existing datasets of paired 2D sketches and photos.
- Domain B: 3D shape datasets (e.g., ShapeNet) and their rendered 2D projections.
By employing a shared 2D encoder across sketches, photos, and projections, the model learns a unified feature space. A KL-divergence loss is applied to align the distribution of distances in the sketch-photo domain with those in the projection-shape domain. This effectively allows the system to "pivot" through the 2D image domain to perform 3D retrieval.
Lifting to 3D Space
To ensure the 2D encoder understands the underlying geometry of the objects—making it robust to viewpoint changes—the authors introduce a "Lifting" module. This module utilizes an auxiliary task: Blind Perspective-n-Points (BPnP).
During training, the model predicts 3D coordinates and confidence weights for sampled 2D points on an image projection. A BPnP solver then estimates the object's pose. The error between the predicted and ground-truth pose is backpropagated to the 2D encoder. This geometric regularization forces the encoder to learn spatial relationships, ensuring that a sketch of a chair from the side is mapped near a 3D model of that same chair, regardless of the sketch's orientation.
Experimental Results
The model was evaluated on several benchmarks, including QMUL-Chair-V2 and ShapeNet, with fine-grained testing on the FG-SBSR dataset.
- Zero-Shot Performance: In a zero-shot setting—where the model never saw a single sketch-3D pair during training—it achieved an Acc@1 of 55.79% on the 'Chairs' category. This performance is remarkably close to the fully supervised "all-shot" baseline of 58.53%, demonstrating the effectiveness of the pivoting strategy.
- Geometric Awareness: The "Lifting" technique provided a significant performance boost over standard 2D encoders. This confirms that injecting "3D common sense" into the feature extraction process is vital for handling the viewpoint variance inherent in free-hand sketching.
- Generalization: On the SHREC’13 category-level retrieval task, the model achieved a Nearest Neighbor (NN) score of 55.3%, proving its utility beyond fine-grained instance retrieval.
Discussion and Limitations
While the results are promising, challenges remain in cross-category generalization. For instance, performance on 'Lamps' (21.63% Acc@1) was lower because the pivot dataset primarily contained chairs. The model's learned "sketch-to-photo" mapping did not transfer perfectly to significantly different geometries.
Future work may involve integrating large-scale foundation models like CLIP to provide a more universal pivot, potentially enabling the system to work across any object category without requiring category-specific paired photo-sketch data.
Conclusion
By reframing 2D-to-3D retrieval as a cross-domain alignment problem mediated by 2D images and geometric lifting, this work provides a scalable path forward. It moves the field away from expensive data collection and toward a more flexible, "democratic" system capable of understanding amateur sketches in a 3D context.
Summary updated