Personalized Sketch-Based Image Retrieval by Convolutional Neural Network and Deep Transfer Learning

Qi Qi, Qiming Huo, Jingyu Wang, Haifeng Sun, Yufei Cao, Jianxin LiaoIEEE Access 2019 · DOI

Introduction

Sketch-Based Image Retrieval (SBIR) allows users to retrieve natural images using simple hand-drawn sketches. Despite its intuitive nature, SBIR faces two primary challenges: the domain gap between sparse sketches and rich natural images, and semantic ambiguity, where a single sketch (e.g., a circle) can represent multiple distinct objects (e.g., a clock or a moon). This research, published in IEEE Access, proposes a personalized retrieval system that leverages deep transfer learning to bridge these gaps by incorporating user intent.

Methodology

The proposed system utilizes a two-branch Convolutional Neural Network (CNN) architecture designed to map both sketches and natural images into a shared feature space.

The General Model

The foundation of the system is a two-branch network based on the VGG-16 architecture. To address the domain gap, natural images are first converted into binary contour maps using the gPb edge detection algorithm. One branch of the CNN processes the hand-drawn sketch, while the other processes the image's contour map. By sharing weights between these branches, the model learns to extract comparable structural features, mapping both inputs into a 512-dimensional vector space. A joint loss function is employed to minimize the distance between sketches and images of the same category while maximizing the distance between different categories.

Personalized Transfer Learning

To resolve semantic ambiguity, the authors introduce a personalized model that adapts to individual user preferences through deep transfer learning.

The two-branch CNN architecture showing fixed layers for feature extraction and fine-tuning layers for personalization.

As illustrated in the architecture above, the first five layers of the pre-trained VGG-16 model are frozen, as they capture universal low-level visual features like lines and edges. The subsequent layers are fine-tuned using a smaller dataset derived from the user's selection history. This process "rearranges" the feature space, moving the user's preferred sub-categories closer to the query sketch. A specialized three-branch loss function is used during fine-tuning to handle positive, general, and negative correlations based on user feedback.

Results and Discussion

The general model was evaluated on the Flickr15K dataset, achieving a Mean Average Precision (mAP) of 0.6449, which outperformed existing Siamese CNN and HOG-based methods.

The impact of personalization was even more significant. By fine-tuning the model on specific user preferences, the system successfully disambiguated sketches. For example, in cases where the general model struggled with overlapping sub-categories, the personalized model achieved an Average Precision (AP) jump from 0.784 to 0.994. T-SNE visualizations confirmed that the fine-tuning process effectively clustered user-preferred sub-categories away from other semantically similar but unintended objects.

Conclusion

By combining a two-branch CNN with deep transfer learning, this approach effectively addresses the core challenges of SBIR. The use of contour maps as a bridge between domains and the application of personalization to resolve semantic ambiguity represent significant steps toward more intuitive and accurate image retrieval systems. Future work will focus on handling noisier user feedback and exploring higher-dimensional data for even more precise intent recognition.

Summary updated

BibTeX
@article{qi2019personalized,
  author = {Qi Qi and Qiming Huo and Jingyu Wang and Haifeng Sun and Yufei Cao and Jianxin Liao},
  title = {{Personalized Sketch-Based Image Retrieval by Convolutional Neural Network and Deep Transfer Learning}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2894351},
}
Figure