Towards Improved and Interpretable Deep Metric Learning via Attentive Grouping

Xinyi Xu, Zhengyang Wang, Cheng Deng, Hao Yuan, Shuiwang Ji

TL;DR

This paper introduces "A-grouping," a novel method for deep metric learning that splits image features into multiple groups using an attention mechanism. This approach not only improves performance by capturing more diverse and discriminative features but also offers unprecedented interpretability, allowing researchers to understand what each feature group is focusing on.

What Problem Does This Paper Address?

Traditional deep metric learning (DML) methods often map images into a single, high-dimensional feature vector. While effective, this can lead to a "dimension saturation" issue where increasing dimensionality yields diminishing returns or even degrades performance. To address this, "deep grouping metric learning" (DGML) emerged, splitting the embedding space into multiple groups, each learning a distinct feature representation. However, existing DGML methods struggle to guarantee that these groups learn truly different and discriminative characteristics, limiting both performance and interpretability. This paper aims to bridge this gap by proposing a DGML method that is both more performant and inherently interpretable.

Methods

The core innovation is the "A-grouping" module. It takes feature maps from a CNN as input and uses an attention mechanism to generate multiple feature groups. Here's how it works:

  1. Key and Value Generation: The input feature maps are processed by 1x1 convolutions to generate "key" and "value" tensors.
  2. Learnable Queries: For each desired group, a unique, learnable "query" vector is introduced. These queries are trained to capture group-specific information.
  3. Attention Weight Maps: The query vectors attend to the key tensors, generating position-wise attention weight maps. These maps indicate the importance of different spatial locations within the feature maps for each specific group.
  4. Group Feature Generation: The attention weight maps are used to compute a weighted sum of the value tensors, producing the final feature vectors for each group.

Crucially, the attention mechanism allows the query to learn which spatial positions are most relevant for its group. This, combined with a diversity loss that encourages different groups to learn distinct features, forms the basis of A-grouping. The method is also designed to be permutation and translation invariant, which is a desirable property for representation learning.

Datasets

The paper evaluates its method on three large-scale image datasets:

  • CUB-200-2011: Contains images of 200 bird categories.
  • Cars-196: Features images from 196 car categories.
  • Stanford Online Products (SOP): A dataset with 120,053 product images across 22,634 categories, noted for its unbalanced class distribution.

Key Insights and Ideas

  • Attention for Interpretability: The attention scores directly translate to spatial importance, providing a clear visual explanation of what each feature group is focusing on. This is a significant step towards making deep learning models more transparent.
  • Group-Specific Queries: Using a learnable query for each group allows the model to tailor its attention mechanism to capture unique characteristics relevant to that specific group.
  • Diversity Loss is Key: The diversity loss is essential for ensuring that different groups learn distinct representations, preventing redundancy and improving overall feature discriminability.
  • Invariance Properties: The formal proof of permutation and translation invariance is a strong theoretical contribution, ensuring robustness to spatial variations in input images.

Key Results

  • Performance Gains: A-grouping consistently outperforms existing DGML methods (like M-grouping, G-grouping, and C-grouping) and even state-of-the-art single-embedding DML methods (like DiVA) across various datasets, backbones (Inception-v1, ResNet-50), and loss functions. For instance, on the CUB-200-2011 dataset, A-grouping with ResNet-50 achieved a Recall@1 of 70.0%, outperforming the next best by a significant margin.
  • Interpretability Demonstrated: Visualizations clearly show that A-grouping groups learn to focus on specific, semantically meaningful parts of objects (e.g., bird body, head, neck, or car windows, headlights, wheels) and are robust to their spatial location. This is a novel and impactful finding for DGML.

Limitations and Future Work

  • Computational Cost: While A-grouping offers significant advantages, the paper notes that DGML methods, in general, can have higher computational costs than traditional DML due to learning multiple embeddings. The SOP dataset experiments were limited due to expensive training times.
  • Hyperparameter Tuning: Like most deep learning methods, A-grouping likely involves hyperparameter tuning, though the paper suggests a relatively simple setup compared to some baselines (e.g., DiVA).
  • Future Work: The paper implicitly suggests exploring more complex attention mechanisms or integrating A-grouping with other advanced DML techniques. Further investigation into the optimal number of groups and their dimensionality for different tasks could also be a future direction.

Summary updated

Abstract

Grouping has been commonly used in deep metric learning for computing diverse features. However, current methods are prone to overfitting and lack interpretability. In this work, we propose an improved and interpretable grouping method to be integrated flexibly with any metric learning framework. Our method is based on the attention mechanism with a learnable query for each group. The query is fully trainable and can capture group-specific information when combined with the diversity loss. An appealing property of our method is that it naturally lends itself interpretability. The attention scores between the learnable query and each spatial position can be interpreted as the importance of that position. We formally show that our proposed grouping method is invariant to spatial permutations of features. When used as a module in convolutional neural networks, our method leads to translational invariance. We conduct comprehensive experiments to evaluate our method. Our quantitative results indicate that the proposed method outperforms prior methods consistently and significantly across different datasets, evaluation metrics, base models, and loss functions. For the first time to the best of our knowledge, our interpretation results clearly demonstrate that the proposed method enables the learning of distinct and diverse features across groups. The code is available on https://github.com/XinyiXuXD/DGML-master.
BibTeX
@article{xu2020towards,
  author = {Xinyi Xu and Zhengyang Wang and Cheng Deng and Hao Yuan and Shuiwang Ji},
  title = {{Towards Improved and Interpretable Deep Metric Learning via Attentive Grouping}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2020},
  url = {https://openreview.net/forum?id=c2JV5D2XGl},
}
Figure