Imagine trying to find a specific 3D model of a chair in a massive database, but instead of typing a text query or uploading another 3D model, you draw a quick, messy sketch on a tablet.
To a human, the connection between a few 2D lines and a dense 3D mesh is obvious. To a computer, however, they are entirely different languages. A 2D sketch is sparse, abstract, and lacks explicit depth or coordinate information. A 3D model (whether represented as a voxel grid, point cloud, or geometric mesh) is dense, coordinate-rich, and highly precise. This structural discrepancy creates a massive cross-modal representation gap.
To bridge this gap, researchers have developed techniques that force these disparate modalities into alignment, allowing algorithms to evaluate shape similarity across dimensions.
Mapping to Shared Latent Spaces
The most direct way to compare a 2D sketch and a 3D shape is to project both into a shared mathematical space where distance corresponds to semantic similarity. If a sketch of a car and a 3D mesh of a car map to nearby coordinates in this latent space, retrieval becomes a simple nearest-neighbor search.
Early approaches focused on joint metric learning. Fan Zhu et al. designed a pyramid cross-domain neural network in Learning Cross-Domain Neural Networks for Sketch-Based 3D Shape Retrieval that aligns features across multiple resolutions, allowing the network to match coarse strokes to global 3D volumes. To ensure the networks "speak the same language," Guoxian Dai et al. introduced a correlated holistic metric learning framework in Deep Correlated Holistic Metric Learning for Sketch-Based 3D Shape Retrieval, training two distinct deep networks with a loss function that explicitly minimizes the statistical distance between their feature distributions.
[2D Sketch] ---> [Sketch Encoder] ---\
---> [Shared Latent Space] (Minimize Distance)
[3D Shape] ---> [3D Encoder] ---/
As databases grow, calculating continuous distances between high-dimensional vectors becomes computationally expensive. To solve this, Jiaxin Chen et al. proposed a binary hashing scheme in Deep Sketch-Shape Hashing With Segmented 3D Stochastic Viewing. By compressing the shared latent space into compact binary codes, they accelerated query times by over 100x while preserving semantic similarity.
However, forcing abstract lines and dense geometry into an arbitrary abstract embedding space can strip away crucial category information. In Semantic Embedding for Sketch-Based 3D Shape Retrieval, Anran Qi et al. argued that instead of an abstract space, models should align in a shared semantic label space. By treating retrieval as a cross-domain classification task, the network learns to map both modalities directly to semantic concepts, significantly stabilizing the alignment process.
Bridging the Gap via 2D Projections and Pivots
Instead of forcing a 3D network to understand 2D lines, why not translate the 3D model into a 2D format? By rendering a 3D shape from multiple angles, we generate a suite of 2D images that can be directly compared to a sketch.
Cunjuan Zhu et al. utilized this perspective in Sketch-Based 3D Shape Retrieval With Multi-View Fusion Transformer, using an attention mechanism to fuse features from different 3D viewpoints. This allows the model to dynamically focus on the specific 3D angle that best matches the user's 2D sketch perspective.
[3D Model] ---> [Multi-View Rendering] ---> [2D Projections] ---\
---> [Transformer Fusion] ---> Alignment
[2D Sketch] ---/
Yet, a rendered 3D view (which contains clean silhouettes or depth maps) still looks fundamentally different from a messy, hand-drawn human sketch. Furthermore, training these models typically requires massive datasets of paired sketches and 3D models, which are expensive to collect.
To democratize this process, Pinaki Nath Chowdhury et al. introduced a "pivoting" strategy in Democratising 2D Sketch to 3D Shape Retrieval Through Pivoting. Instead of training on direct sketch-to-3D pairs, they used natural 2D photographs as an intermediate bridge. Because large datasets already exist for sketch-to-photo and photo-to-3D relationships, the photo domain acts as a universal translator, allowing the model to lift sketches into 3D space via geometric constraints without direct supervision.
This concept of leveraging pre-existing knowledge was pushed even further by Gianluca Berardi in Fine-Tuned but Zero-Shot 3D Shape Sketch View Similarity and Retrieval. The author demonstrated that vision-language foundation models like CLIP already possess a deep understanding of cross-modal abstractions. By fine-tuning CLIP on synthetic views of just a single object category, the model's internal representation generalizes zero-shot across completely unseen object types and diverse artistic styles.
Disentangling Structure from Style
A major reason cross-modal alignment fails is that neural networks easily get distracted by domain-specific details. A 2D photograph contains lighting, shadows, and complex backgrounds; a sketch contains unique artistic line styles; a 3D CAD model contains pure, unadorned geometry. If a network focuses on these stylistic elements, it fails to match objects based on their actual shape.
To isolate pure geometry, researchers use feature disentanglement and adversarial training. In Domain Disentangled Generative Adversarial Network for Zero-Shot Sketch-Based 3D Shape Retrieval, Rui Xu et al. used a GAN to split features into two distinct components: a domain-invariant semantic representation (the "what") and a domain-specific style representation (the "how"). By discarding the style component during retrieval, the system matches sketches to 3D models purely on their shared semantic structure.
[Input Feature] ---> [Disentanglement GAN] ---> [Domain-Specific Style] (Discarded)
---> [Domain-Invariant Geometry] (Used for Retrieval)
Conversely, when retrieving 3D models using feature-rich 2D photos, the challenge is forcing the network to ignore distracting surface textures. Huan Fu et al. solved this in Hard Example Generation by Texture Synthesis for Cross-domain Shape Similarity Learning by intentionally painting complex textures onto the wrong 3D shapes during training. This creates "hard triplets" that trick the network; to succeed, the model is forced to ignore the deceptive surface colors and focus strictly on underlying geometric similarity.
For hand-drawn inputs, another way to bypass surface noise entirely is to focus on topology. Jing Zhang et al. proposed a structural approach in A novel sketch-based 3D model retrieval approach based on skeleton, extracting the topological "skeleton" or backbone of both the 2D sketch and the 3D model. By matching these simplified structural graphs, the system becomes highly robust to line deformations and user drawing imperfections.
Robustness and Unsupervised Adaptation
In real-world applications, data is rarely perfectly curated. Training sets often contain mislabeled pairs, and models must frequently operate in semi-supervised or unsupervised environments where labeled 3D data is unavailable.
To handle real-world data imperfections, Yanglin Feng et al. developed RONO in RONO: Robust Discriminative Learning with Noisy Labels for 2D-3D Cross-Modal Retrieval. By exploiting the "memorization effect" of deep networks—the tendency for models to learn clean patterns before memorizing noise—their framework filters out incorrectly paired 2D-3D training data, ensuring stable cross-modal alignment even under heavy label corruption.
When labeled 3D data is entirely absent, the problem transitions into unsupervised domain adaptation. Heyu Zhou et al. introduced HIFA in Hierarchical Instance Feature Alignment for 2D Image-Based 3D Shape Retrieval, which maximizes mutual information between domains using a two-tier alignment strategy that matches global feature distributions alongside local class centroids.
Complementing this, Yuting Su et al. developed a framework called CDSL in Consistent Domain Structure Learning and Domain Alignment for 2D Image-Based 3D Objects Retrieval. CDSL enforces a logical constraint: the predicted category structures in the 2D image domain must remain mathematically consistent with the natural geometric clusters found within the unlabelled 3D data.
Finally, when labeled data is scarce, Fan Zhang et al. demonstrated in Fine-grained Prototypical Voting with Heterogeneous Mixup for Semi-supervised 2D-3D Cross-modal Retrieval that maintaining multiple "prototypes" per object class allows a network to vote on unlabeled data. By mixing features directly across the heterogeneous boundaries of 2D pixels and 3D points, the model smoothly interpolates the representation gap.
Beyond Retrieval: Generative Workflows
While cross-modal alignment is traditionally used for retrieval, the underlying mechanics are increasingly being applied to generative engineering pipelines.
Moving past simple shape matching, Feiwei Qin demonstrated in Drawing2CAD: Sequence-to-Sequence Learning for CAD Generation from Vector Drawings how to translate 2D vector engineering drawings (SVGs) directly into 3D parametric CAD models. By utilizing a dual-decoder Transformer architecture, this approach bypasses retrieval entirely, translating the abstract, sequential tokens of a 2D vector drawing directly into the structural commands needed to construct a 3D physical object.
Key Foundational Work
A comparison of 3D shape retrieval methods based on a large-scale benchmark supporting multimodal queries (2015) — contributed 3 concept elements to this direction.
Learning Cross-Domain Neural Networks for Sketch-Based 3D Shape Retrieval (2016) — contributed 1 concept elements to this direction.