s-SBIR: Style Augmented Sketch based Image Retrieval

Titir Dutta, Soma BiswasWACV 2020

TL;DR

This paper introduces "style-augmented sketch-based image retrieval" (s-SBIR), a new problem where users can retrieve images not only by sketch but also by specifying desired visual styles (e.g., color, shape). They propose a novel deep learning framework with separate modules for category, style, and a "mixer" to combine these, along with a new evaluation metric (ncMAP) to better assess performance.

What Problem Does This Paper Address?

Traditional sketch-based image retrieval (SBIR) focuses on finding natural images that match the structural content of a hand-drawn sketch. However, users often have preferences beyond just the object's shape, such as its color, texture, or material. Existing SBIR methods don't account for these stylistic attributes, leading to retrieved images that might be structurally correct but stylistically unsuitable. This paper addresses this gap by introducing s-SBIR, enabling users to specify both the object category via a sketch and desired styles using keywords.

Methods

The core of the proposed approach is a novel end-to-end deep framework with three main components:

  • Category Module: This module extracts content features from both sketches and images using fine-tuned VGG-19 networks. It then learns a shared latent space (Φ-space) using an encoder-decoder structure, mapping domain-specific features to a common representation. This space is designed to preserve semantic relationships between categories, leveraging GloVe embeddings for category names.
  • Style Module: This module handles the extraction of style information. For keywords provided by the user, it uses GloVe embeddings of the style terms and passes them through a style encoder. For images, it extracts style features from the activation maps of early convolutional layers (using Gram matrices) and then encodes them. The goal is to learn style representations that can be directly compared.
  • Mixer Module: This crucial component takes the encoded category and style information (from both sketch queries and database images) and fuses them into a composite representation in the Φ-space. This module is trained using a quintuplet loss that enforces that images matching both category and style are closer to the query than those matching only one or neither.

The paper also introduces a new evaluation metric, normalized composite Mean Average Precision (ncMAP), which combines category and style scores to provide a more comprehensive assessment of s-SBIR performance, reducing to standard MAP when style is ignored.

Datasets

The authors use modified versions of two popular sketch-based image retrieval datasets:

  • m-Sketchy: An extension of the original Sketchy dataset, containing 31,378 sketches and 31,900 images from 53 categories.
  • m-TU-Berlin: An extension of the TU-Berlin dataset, with 7600 sketches and 8587 images from 95 classes, augmented with additional natural images.

These datasets are modified to include instance-level style annotations, which were not fully available in the original versions.

Key Insights and Ideas

  • Style as a First-Class Citizen: The paper elevates style from a secondary consideration to a primary search criterion alongside category, acknowledging its importance in real-world image retrieval.
  • Disentangling Content and Style: The framework effectively separates content (object shape) from style (color, texture, etc.) using distinct modules, allowing for independent processing and then intelligent fusion.
  • GloVe for Semantic Style Representation: Using GloVe embeddings for style keywords, rather than simple binary representations, allows for capturing semantic relationships between styles (e.g., "red" being closer to "pink" than "black"), leading to more nuanced retrieval.
  • Φ-space for Unified Representation: The shared latent space (Φ-space) is key to enabling cross-modal comparison between sketches and images, and the mixer module further refines this space for style-aware retrieval.
  • Need for a New Metric: The inadequacy of standard MAP for style-aware retrieval is highlighted, motivating the development of ncMAP to provide a more accurate evaluation.

Key Results

  • The proposed s-SBIR framework achieves state-of-the-art performance on the m-Sketchy and m-TU-Berlin datasets, outperforming several baselines.
  • The ncMAP metric shows significant improvements over standard MAP, particularly for baselines that do not adequately consider style.
  • Qualitative results demonstrate that the framework successfully retrieves images that match both the sketch category and the specified styles, pushing stylistically relevant images to the top.
  • Experiments show that using GloVe embeddings for style keywords leads to perceptually similar style matches, even when the exact style is not present in the dataset.

Limitations and Future Work

  • Dataset Limitations: The authors acknowledge the unavailability of large-scale datasets with comprehensive style annotations, which necessitates creating modified datasets.
  • Style Representation: While GloVe embeddings are effective, future work could explore more sophisticated methods for representing and learning style.
  • Multiple Styles: The framework currently handles single or two styles effectively. Extending it to seamlessly manage a larger number of style constraints is a potential future direction.
  • Generalization: While the framework shows promise, further evaluation on diverse datasets and style types would be beneficial.

Summary updated

BibTeX
@inproceedings{dutta2020s,
  author = {Titir Dutta and Soma Biswas},
  title = {{s-SBIR: Style Augmented Sketch based Image Retrieval}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure