Uncertainty-Aware Cross-Modal Transfer Network for Sketch-Based 3D Shape Retrieval

Yiyang Cai, Jiaming Lu, Jiewen Wang, Shuang Liang

Abstract

Sketch-Based 3D Shape Retrieval (SBSR) remains a challenging task due to the significant modality gap between 2D sketches and 3D models, compounded by the inherent noise and abstraction in human drawings. This paper presents the Uncertainty-Aware Cross-Modal Transfer Network (UACTN), a novel framework that addresses these challenges by modeling sketches as probabilistic distributions rather than deterministic points. By decoupling the learning process and incorporating Classification-Based Uncertainty Learning (CBUL), UACTN effectively mitigates the impact of low-quality queries. Experimental results on the SHREC 2013 and 2014 benchmarks demonstrate that UACTN achieves state-of-the-art performance, offering a robust solution for noise-tolerant cross-modal retrieval.

Introduction

Sketch-Based 3D Shape Retrieval (SBSR) provides an intuitive interface for users to query 3D databases using simple 2D drawings. However, the practical application of SBSR is hindered by two primary obstacles: the modality gap and data noise. The modality gap arises from the fundamental difference between abstract 2D line drawings and dense 3D geometric volumes. Furthermore, human sketches are often highly variable; some are so poorly executed or abstract that they contain little discriminative information.

Standard retrieval models often suffer from overfitting when presented with such "noisy" data, as they attempt to map every input—regardless of quality—to a fixed point in a shared embedding space. Previous methods, such as the Sketch-based Uncertainty Learning (SUL) approach, attempted to address this but were limited by multi-step training procedures that prevented end-to-end optimization. The UACTN framework overcomes these limitations by treating sketch features as Gaussian distributions and utilizing a decoupled transfer learning strategy.

Methodology

The UACTN framework introduces two core innovations designed to enhance robustness and alignment across modalities.

1. Classification-Based Uncertainty Learning (CBUL)

To handle the variability of human sketches, UACTN represents each sketch as a Gaussian distribution defined by a mean ($\mu$) and a variance ($\sigma$). The mean represents the core semantic features of the sketch, while the variance captures the level of uncertainty or noise.

During training, the model employs the "reparameterization trick" to sample from this distribution. This probabilistic approach acts as a natural regularizer: when a sketch is noisy or poorly drawn, the model predicts a high variance. In the context of the Large Margin Cosine Loss, this high variance makes the sample's position in the embedding space more "random," preventing the network from forcing a fit to irrelevant or misleading details. Consequently, the model focuses its learning capacity on high-quality, recognizable samples.

2. Decoupled 3D Shape Feature Transfer

Simultaneous training of sketch and 3D shape branches often leads to an imbalance where one modality dominates the learning process, failing to bridge the modality gap effectively. UACTN addresses this through a two-stage decoupled process:

  • Stage 1: The sketch branch is trained independently to establish a robust, noise-resistant embedding space with fixed class centers.
  • Stage 2: 3D shape features are "transferred" into this pre-established sketch space. By mapping 3D features to align with the stable class centers defined by the sketches, the model ensures better cross-modal alignment without the instability of joint training.

Experimental Evaluation

The UACTN framework was evaluated on two gold-standard benchmarks: SHREC 2013 (7,200 sketches, 1,258 shapes) and SHREC 2014 (13,680 sketches, 8,987 shapes). The latter is particularly challenging due to its larger scale and the presence of semantically similar categories.

Performance Metrics

UACTN demonstrated superior performance across all standard metrics, including Nearest Neighbor (NN), First Tier (FT), and Mean Average Precision (mAP). On the SHREC 2014 dataset using a ResNet50 backbone, UACTN achieved a mAP of 84.8%, outperforming previous state-of-the-art methods like SUL (83.4%) and JFLN (83.3%).

Qualitative Analysis

Visualizations of the uncertainty mapping confirm the model's efficacy. Clear, detailed drawings (e.g., a recognizable airplane) are assigned low uncertainty values, while ambiguous scribbles are assigned high uncertainty. This confirms that the model successfully identifies and down-weights "garbage" input data during the retrieval process.

Discussion and Limitations

A key insight of this work is that uncertainty can serve as a powerful regularizer in cross-modal tasks. By embedding uncertainty directly into the feature representation rather than using it as a simple loss weight, the entire network becomes optimized for noise robustness from the ground up.

However, the model currently faces a limitation regarding "unusual viewpoints." Sketches that are well-drawn but depict an object from a rare perspective are sometimes misclassified as high-uncertainty (noisy) samples. This occurs because the model conflates statistical rarity with low quality. Future research will focus on distinguishing between truly noisy inputs and "hard samples" (clean but rare perspectives) to further refine retrieval precision.

Conclusion

The Uncertainty-Aware Cross-Modal Transfer Network (UACTN) represents a significant advancement in Sketch-Based 3D Shape Retrieval. By treating sketches as probabilistic distributions and decoupling the cross-modal learning process, the framework achieves new levels of robustness against the inherent noise of human drawings. The results on SHREC benchmarks validate UACTN as a state-of-the-art approach for bridging the gap between 2D human creativity and 3D digital geometry.

Summary updated

BibTeX
@inproceedings{cai2023uncertainty,
  author = {Yiyang Cai and Jiaming Lu and Jiewen Wang and Shuang Liang},
  title = {{Uncertainty-Aware Cross-Modal Transfer Network for Sketch-Based 3D Shape Retrieval}},
  booktitle = {ICME},
  year = {2023},
  doi = {10.1109/icme55011.2023.00031},
}
Figure