Coded Residual Transform for Generalizable Deep Metric Learning

Shichao Kan, Yixiong Liang, Min Li, Yigang Cen, Jianxin Wang, Zhihai He

Deep Metric Learning (DML) aims to learn a feature embedding space where semantically similar images are mapped to nearby points while dissimilar ones are pushed apart. Despite significant progress, a persistent "generalization gap" remains: models often fail to maintain performance when encountering unseen object categories. In their NeurIPS 2022 work, Kan et al. propose the Coded Residual Transform (CRT) to address this by shifting from absolute feature encoding to relative residual transforms across multiple perspectives.

The Generalization Challenge in DML

Standard DML approaches typically project images into an embedding space from a single global perspective. This often collapses complex high-dimensional correlation structures, leading to high marginal variance and poor generalization to novel classes. Furthermore, these models often rely on absolute feature values, which are sensitive to environmental noise and domain shifts. The CRT framework aims to capture invariant relative patterns, much like how humans describe objects by their differences from a known "prototype."

The Coded Residual Transform (CRT) Framework

The CRT framework addresses these limitations by encoding feature maps as weighted residuals relative to a set of learned, diversified prototypes.

The Coded Residual Transform (CRT) architecture, illustrating the flow from input image to nonlinear embedding via multi-perspective projection and residual encoding.

As shown in the architecture above, the system consists of three primary components:

  1. Multi-Perspective Projection: The model learns $K$ diversified prototypes. The input feature map is projected onto each, providing complementary views of the data.
  2. Correlation-Based Residual Encoding: For each prototype, the model computes the residual—the difference between local features and the prototype—weighted by their mutual correlation. This ensures that only relevant prototypes influence specific spatial regions of the embedding.
  3. Cross-CRT Consistency Loss: To enhance robustness, two parallel CRT branches with varying prototype counts and embedding dimensions are trained. A consistency loss is enforced on their similarity matrices, acting as a scale-invariant regularizer.

Multi-Perspective Insights

A key insight of CRT is that relative differences (residuals) are more stable across domains than raw feature values. By forcing the learned prototypes to be diverse through a specific diversity loss, the model is compelled to analyze the data from multiple unique "angles."

Visualizing multi-perspective attention: different prototypes focus on distinct semantic parts of the object, capturing a richer set of invariant features.

Visualization of the attention maps reveals that different perspectives attend to different parts of the object (e.g., the head, body, or tail of a bird). This multi-faceted representation captures invariant patterns that transfer more effectively to new classes. Spectral decay analysis further confirms that CRT promotes a more uniform distribution of information across embedding dimensions, preventing over-reliance on a few dominant features that might not generalize.

Experimental Results

The authors evaluated CRT on four standard benchmarks: CUB-200-2011, Cars-196, Stanford Online Products (SOP), and In-Shop Clothes Retrieval. CRT demonstrated substantial improvements across all datasets:

  • CUB-200-2011: Achieved a 78.98% Recall@1, representing a 4.28% improvement over the previous state-of-the-art.
  • In-Shop: Improved Recall@1 by 1.68%, reaching 94.48%.
  • Backbone Agnostic: The performance gains were consistent across various architectures, including GoogLeNet and Vision Transformers (ViT/DeiT).

Conclusion

The Coded Residual Transform offers a robust solution to the generalization gap in deep metric learning. By leveraging multi-perspective residual encoding and cross-branch consistency, it captures stable, relative patterns that are invariant to the noise of absolute feature values. While it introduces some computational overhead during training, its significant performance gains and backbone-agnostic nature make it a powerful tool for modern retrieval tasks.

Summary updated

Abstract

A fundamental challenge in deep metric learning is the generalization capability of the feature embedding network model since the embedding network learned on training classes need to be evaluated on new test classes. To address this challenge, in this paper, we introduce a new method called coded residual transform (CRT) for deep metric learning to significantly improve its generalization capability. Specifically, we learn a set of diversified prototype features, project the feature map onto each prototype, and then encode its features using their projection residuals weighted by their correlation coefficients with each prototype. The proposed CRT method has the following two unique characteristics. First, it represents and encodes the feature map from a set of complimentary perspectives based on projections onto diversified prototypes. Second, unlike existing transformer-based feature representation approaches which encode the original values of features based on global correlation analysis, the proposed coded residual transform encodes the relative differences between the original features and their projected prototypes. Embedding space density and spectral decay analysis show that this multi perspective projection onto diversified prototypes and coded residual representation are able to achieve significantly improved generalization capability in metric learning. Finally, to further enhance the generalization performance, we propose to enforce the consistency on their feature similarity matrices between coded residual transforms with different sizes of projection prototypes and embedding dimensions. Our extensive experimental results and ablation studies demonstrate that the proposed CRT method outperform the state-of-the-art deep metric learning methods by large margins and improving upon the current best method by up to 4.28% on the CUB dataset.
BibTeX
@inproceedings{kan2022coded,
  author = {SHICHAO KAN and Yixiong Liang and Min Li and Yigang Cen and Jianxin Wang and Zhihai He},
  title = {{Coded Residual Transform for Generalizable Deep Metric Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=AlgbeSuE1lx},
}
Figure