TL;DR
This paper introduces a novel "Stacked Semantic-Guided Network" for Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR). It tackles the challenges of sketch sparsity and intra-class variance by employing a multi-layer feature fusion approach and a coarse-to-fine conditional decoder, significantly outperforming existing methods.
What Problem Does This Paper Address?
Sketch-Based Image Retrieval (SBIR) allows users to search for images using free-hand sketches. However, a more challenging variant, Zero-Shot SBIR (ZS-SBIR), aims to retrieve images for categories not seen during training. This is particularly difficult because sketches are inherently sparse and exhibit large variations in how people draw them (e.g., different levels of detail or abstraction). Existing ZS-SBIR methods often struggle to transfer knowledge from seen to unseen classes due to these sketch characteristics, leading to unsatisfactory retrieval performance. This paper addresses this gap by proposing a method that explicitly handles the unique properties of sketches.
Methods
The proposed "Stacked Semantic-Guided Network" has several key components:
- Multi-Layer Feature Fusion Networks: To combat sketch sparsity, the method fuses features from multiple intermediate layers of a pre-trained deep neural network (like VGG16). This enriches the sketch representation without adding extra parameters, capturing information at different levels of abstraction. Principal Component Analysis (PCA) is used for dimensionality reduction.
- Conditional Feature Encoder: A Conditional Variational Autoencoder (CVAE) is employed to learn the mapping from sketches and semantic information to latent representations.
- Coarse-to-Fine Conditional Decoder: This is the core innovation for handling intra-class variance. It generates synthesized image features in two stages:
- Coarse-grained generation: Uses category-level semantic information (like word vectors) to generate initial image features. This leverages the low intra-class variance of semantic information.
- Fine-grained generation: Uses the coarse features and the sketch itself to generate more detailed, instance-specific image features. This progressive approach makes learning more efficient and improves knowledge transfer.
- Semantic-Preserving Network: A regressor is used to ensure that the synthesized features maintain semantic relationships with the training categories.
- Discriminability-Preserving Network: A classifier is used to ensure that the synthesized features are discriminative, helping to distinguish between different categories.
The entire model is optimized using a combination of KL divergence loss, reconstruction loss, regression loss, and classification loss.
Datasets
The paper evaluates its method on two widely used large-scale datasets:
- Sketchy: Originally 75,479 sketches and 12,500 images across 125 categories. The dataset was extended with additional images from ImageNet. For ZS-SBIR, the 125 categories are split into 104 training (seen) and 21 testing (unseen) categories.
- TU-Berlin: Consists of 20,000 sketches across 250 categories. An extended version with 204,070 natural images is used. The categories are split into 165 training and 85 testing categories.
Key Insights and Ideas
- Multi-layer fusion is crucial for sketches: Simply using the final layer features of a CNN is insufficient for sketches. Fusing features from multiple layers effectively captures richer, more robust representations.
- Coarse-to-fine generation is key for handling variance: Directly generating fine-grained features from sparse sketches is difficult. A two-stage approach, starting with semantic information and then refining with sketch details, significantly improves performance and learning stability.
- Semantic information complements sketch features: While sketches provide detailed shape information, semantic embeddings offer a more stable, low-variance representation of categories, which is beneficial for initial generation.
- Regularization for semantic and discriminative features: Explicitly enforcing semantic preservation and discriminability on generated features leads to better retrieval performance.
Key Results
The proposed method significantly outperforms existing state-of-the-art methods on both the Sketchy and TU-Berlin datasets, achieving improvements of over 20% in retrieval performance (measured by mAP@200). For instance, on Sketchy, the proposed method (MLFF+C2F+DP) achieves an mAP@200 of 0.479, compared to 0.225 for CVAE. On TU-Berlin, it achieves 0.320 mAP@200, compared to 0.104 for CVAE. Visualizations (confusion matrices and T-SNE plots) also demonstrate better class separation and more compact feature representations for the proposed method.
Limitations and Future Work
- Fixed Feature Extractor: The current model uses pre-trained VGG16 features. The authors suggest that an end-to-end joint feature extraction and retrieval approach could further boost performance.
- Semantic Information Availability: While semantic information is used for training, its availability during testing is assumed. Future work could explore scenarios where only category-level word vectors are available at training time.
- Failure Cases: The confusion matrices reveal that some categories are still challenging to retrieve accurately. Future research should focus on better understanding and addressing these failure cases.
Summary updated