Domain-Aware SE Network for Sketch-based Image Retrieval with Multiplicative Euclidean Margin Softmax

Peng Lu, Gao Huang, Hangyu Lin, Wenming Yang, Guodong Guo, Yanwei Fu

TL;DR

This paper introduces a novel approach for Sketch-Based Image Retrieval (SBIR) by developing a "Domain-Aware Squeeze-and-Excitation" (DASE) network that explicitly leverages domain knowledge (sketch vs. photo) to improve feature extraction. They also propose a new loss function, "Multiplicative Euclidean Margin Softmax" (MEMS), which creates a more discriminative feature space by enforcing a larger separation between classes than previously achieved.

What Problem Does This Paper Address?

Sketch-Based Image Retrieval (SBIR) is the task of finding relevant photos given a query sketch. The core challenge lies in bridging the significant "domain gap" between the abstract, line-based representation of sketches and the dense, pixel-rich representation of photographs. Existing methods often struggle to effectively learn shared semantic representations across these domains, leading to suboptimal retrieval accuracy. While some approaches try to transform one domain into the other (e.g., generating edge maps from photos or photos from sketches), these can be computationally expensive or introduce artifacts. The authors aim to create a unified feature space that directly handles both sketches and photos, making the retrieval process more efficient and accurate.

Methods

The paper proposes two main innovations:

  • Domain-Aware Squeeze-and-Excitation (DASE) Network: This is an extension of the standard Squeeze-and-Excitation (SE) module, which is known for adaptively recalibrating channel-wise feature responses. The DASE module incorporates a binary code indicating whether the input is a sketch or a photo. This domain information is fed into the SE module, allowing it to learn domain-specific channel attention. This means the network can emphasize different features for sketches versus photos, leading to more robust and discriminative representations. The DASE module is integrated into each residual block (ResBlock) of a shared Siamese-like network architecture that processes both sketches and photos.
  • Multiplicative Euclidean Margin Softmax (MEMS) Loss: Traditional loss functions for metric learning aim to minimize intra-class distance and maximize inter-class distance. However, they often lack a guaranteed margin or are difficult to optimize. The MEMS loss introduces a multiplicative Euclidean margin. It aims to ensure that the maximum intra-class distance is strictly smaller than the minimum inter-class distance, and specifically, it enforces that the intra-class distance is at most $1/m$ times the inter-class distance, where $m$ is a margin constant. This multiplicative margin is designed to create a highly discriminative feature space. The authors provide theoretical analysis showing that $m \ge 2 + \sqrt{3}$ is a sufficient and necessary condition for perfect discrimination in the binary case, and this holds for multi-class scenarios as well. The MEMS loss is optimized using back-propagation and is shown to be more stable and effective than previous approaches.

Datasets

The proposed method is evaluated on two widely used large-scale sketch-photo datasets:

  • TU-Berlin Extension: Contains 20,000 sketches across 250 categories and 204,489 natural images for the gallery.
  • Sketchy Extension: Consists of 75,471 sketches and 12,500 corresponding photos from 125 categories, extended with an additional 60,502 photos for the retrieval task.

The authors follow standard splits for training and testing, using a portion of sketches for queries and the rest for training, with all gallery photos used for both training and testing.

Key Insights and Ideas

  • Explicit Domain Knowledge is Powerful: The DASE network demonstrates that simply informing the model about the input domain (sketch vs. photo) can significantly improve its ability to learn discriminative features. This is a more direct approach than domain translation methods.
  • Multiplicative Margin for Discriminative Features: The MEMS loss offers a novel way to enforce a strong separation between classes. By using a multiplicative margin, it ensures a more robust and compact feature space compared to additive margins, which can be less effective when distances are large.
  • Theoretical Guarantees for Discrimination: The theoretical analysis of the MEMS loss provides confidence in its ability to create a well-separated feature space, with a clear mathematical condition ($m \ge 2 + \sqrt{3}$) for achieving perfect discrimination.
  • Unified Framework for SBIR and Hashing: The proposed DASE network and MEMS loss can be used to generate real-valued features for retrieval or can be adapted for SBIR hashing, a technique that significantly speeds up retrieval by converting features into binary codes. The hashing scheme preserves the discriminative properties learned by the MEMS loss.

Key Results

  • State-of-the-Art Performance: The proposed DASE+MEMS approach achieves state-of-the-art results on both the TU-Berlin Extension and Sketchy Extension datasets, outperforming existing methods by a significant margin. For instance, on TU-Berlin Extension, it achieves a Mean Average Precision (MAP) of 0.841, and on Sketchy Extension, it reaches 0.958.
  • Improved Hashing Performance: In SBIR hashing tasks, the DASE+MEMS model also sets new benchmarks, outperforming methods like GDH by over 0.13 MAP. This highlights the effectiveness of the learned features for dimension reduction and binary encoding.
  • Efficiency: The DASE network can be trained efficiently in an end-to-end manner without domain translation. The hashing scheme is also very fast, taking only about a minute to train after feature extraction.
  • Ablation Studies: Experiments confirm that both the DASE module and the MEMS loss contribute significantly to the performance gains. The DASE module consistently improves upon standard SE modules. The choice of the margin $m$ in MEMS loss is crucial, with $m=4$ showing optimal performance in their experiments.

Limitations and Future Work

  • Margin Selection: While the theoretical analysis provides a lower bound for $m$, the optimal value of $m$ still needs to be determined empirically. The paper notes that performance can degrade if $m$ is too large.
  • Hashing Network Depth: The paper suggests that a single-layer linear mapping (FSN) for hashing works best, outperforming multi-layer networks. This might indicate that complex hashing architectures are not always necessary or could even be detrimental when combined with well-learned features.
  • Future Work: The authors suggest exploring more sophisticated hashing schemes that can better leverage the learned discriminative features. They also imply that the DASE module and MEMS loss could be applicable to other cross-modal retrieval tasks beyond SBIR.

Summary updated

Abstract

This paper proposes a novel approach for Sketch-Based Image Retrieval (SBIR), for which the key is to bridge the gap between sketches and photos in terms of the data representation. Inspired by channel-wise attention explored in recent years, we present a Domain-Aware Squeeze-and-Excitation (DASE) network, which seamlessly incorporates the prior knowledge of sample sketch or photo into SE module and make the SE module capable of emphasizing appropriate channels according to domain signal. Accordingly, the proposed network can switch its mode to achieve a better domain feature with lower intra-class discrepancy. Moreover, while previous works simply focus on minimizing intra-class distance and maximizing inter-class distance, we introduce a loss function, named Multiplicative Euclidean Margin Softmax (MEMS), which introduces multiplicative Euclidean margin into feature space and ensure that the maximum intra-class distance is smaller than the minimum inter-class distance. This facilitates learning a highly discriminative feature space and ensures a more accurate image retrieval result. Extensive experiments are conducted on two widely used SBIR benchmark datasets. Our approach achieves better results on both datasets, surpassing the state-of-the-art methods by a large margin.
BibTeX
@inproceedings{lu2021domain,
  author = {Peng Lu and Gao Huang and Hangyu Lin and Wenming Yang and Guodong Guo and Yanwei Fu},
  title = {{Domain-Aware SE Network for Sketch-based Image Retrieval with Multiplicative Euclidean Margin Softmax}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475499},
}
Figure