StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval

Aneeshan Sain, Ayan Kumar Bhunia, Yongxin Yang, Tao Xiang, Yi-Zhe Song

Abstract

Sketch-based image retrieval (SBIR) is a cross-modal matching problem which is typically solved by learning a joint embedding space where the semantic content shared between photo and sketch modalities are preserved. However, a fundamental challenge in SBIR has been largely ignored so far, that is, sketches are drawn by humans and considerable style variations exist amongst different users. An effective SBIR model needs to explicitly account for this style diversity, crucially, to generalise to unseen user styles. To this end, a novel style-agnostic SBIR model is proposed. Different from existing models, a cross-modal variational autoencoder (VAE) is employed to explicitly disentangle each sketch into a semantic content part shared with the corresponding photo, and a style part unique to the sketcher. Importantly, to make our model dynamically adaptable to any unseen user styles, we propose to meta-train our cross-modal VAE by adding two style-adaptive components: a set of feature transformation layers to its encoder and a regulariser to the disentangled semantic content latent code. With this meta-learning framework, our model can not only disentangle the cross-modal shared semantic content for SBIR, but can adapt the disentanglement to any unseen user style as well, making the SBIR model truly style-agnostic. Extensive experiments show that our style-agnostic model yields state-of-the-art performance for both category-level and instance-level SBIR.

What Problem Does This Paper Address?

Sketch-Based Image Retrieval (SBIR) aims to find a specific photo based on a hand-drawn sketch. While researchers have made great strides in bridging the "domain gap" (the difference between a pixel-perfect photo and a sparse line drawing), they have largely ignored the "style gap."

Human sketching is subjective. One person might focus on the silhouette of a shoe, while another focuses on the laces or the heel. Existing models typically try to learn a joint embedding space that ignores these variations, but they often fail when a new user provides a sketch in a style the model hasn't encountered. The authors argue that an effective SBIR model must be "style-agnostic"—it needs to explicitly model and then set aside stylistic quirks to find the underlying semantic content.

StyleMeUp handles diverse sketching styles for the same object, whereas existing models often fail when encountering unseen user styles.

Methods

The authors introduce a framework that combines disentangled representation learning with meta-learning to create a model that can adapt to any user's drawing style on the fly.

1. Cross-Modal VAE

The core architecture is a Variational Autoencoder. The encoder is designed to decompose an input (sketch or photo) into two distinct latent codes:

  • Modal-Invariant Semantic Part: The "content" shared between a photo and its corresponding sketches.
  • Modal-Specific Style Part: The "style" unique to the sketcher (or the specific characteristics of the photo).

2. Meta-Learning (MAML)

To ensure the model can adapt to unseen styles without retraining, the authors use a Model-Agnostic Meta-Learning (MAML) framework. The model is trained across "episodes" of different sketching styles, learning how to quickly adapt its disentanglement process.

3. Style-Adaptive Components

They introduce two new modules into the meta-learning pipeline, as shown in Figure 2:

  • Feature Transformation (FT) Layers: Plugged into the encoder to dynamically minimize style variance.
  • Meta-Regularizer: A learnable $\ell_1$ regularizer that adaptively modulates how much semantic knowledge is retained, ensuring the disentanglement is effective even for new categories or styles.
The StyleMeUp architecture uses a meta-learned cross-modal VAE to disentangle semantic content from stylistic variations through inner and outer loops.

Datasets and Evaluation

The authors evaluated their model on four major benchmarks, covering both category-level and fine-grained (instance-level) retrieval:

  • Category-Level: Sketchy (extended) and TU-Berlin (extended).
  • Fine-Grained (Instance-Level): QMUL-Chair-V2 and QMUL-Shoe-V2. These are critical because they contain multiple sketches of the same object instance drawn by different people, allowing the model to practice "ignoring" style to find the specific object.

Key Results

StyleMeUp consistently outperformed existing state-of-the-art (SOTA) methods across all datasets:

  • Fine-Grained SBIR: On the QMUL-Shoe-V2 dataset, it achieved 36.47% acc@1, a significant jump over previous SOTA methods like Triplet-RL (34.10%) and CC-Gen (33.80%).
  • Category-Level SBIR: On the Sketchy dataset, it reached a 0.905 mAP, compared to 0.810 for the previous best (GDH).
  • Consistency: The model showed much lower "rank variance." This means that regardless of which user drew the sketch, the model's retrieval performance remained stable, whereas baseline models fluctuated wildly depending on the sketcher's style.

Conclusion

By treating style as a structured distractor rather than random noise, StyleMeUp moves SBIR toward real-world usability. The model's ability to perform dynamic disentanglement on unseen user styles makes it truly style-agnostic. Future work may extend this approach to other cross-modal tasks, such as text-to-image synthesis or forensic sketch matching, where subjective interpretation is a major factor.

Summary updated

Abstract

Sketch-based image retrieval (SBIR) is a cross-modal matching problem which is typically solved by learning a joint embedding space where the semantic content shared between photo and sketch modalities are preserved. However, a fundamental challenge in SBIR has been largely ignored so far, that is, sketches are drawn by humans and considerable style variations exist amongst different users. An effective SBIR model needs to explicitly account for this style diversity, crucially, to generalise to unseen user styles. To this end, a novel style-agnostic SBIR model is proposed. Different from existing models, a cross-modal variational autoencoder (VAE) is employed to explicitly disentangle each sketch into a semantic content part shared with the corresponding photo, and a style part unique to the sketcher. Importantly, to make our model dynamically adaptable to any unseen user styles, we propose to meta-train our cross-modal VAE by adding two style-adaptive components: a set of feature transformation layers to its encoder and a regulariser to the disentangled semantic content latent code. With this meta-learning framework, our model can not only disentangle the cross-modal shared semantic content for SBIR, but can adapt the disentanglement to any unseen user style as well, making the SBIR model truly style-agnostic. Extensive experiments show that our style-agnostic model yields state-of-the-art performance for both category-level and instance-level SBIR.
BibTeX
@inproceedings{sain2021stylemeup,
  author = {Aneeshan Sain and Ayan Kumar Bhunia and Yongxin Yang and Tao Xiang and Yi-Zhe Song},
  title = {{StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00840},
}
Figure