Moving Towards Centers: Re-ranking with Attention and Memory for Re-identification

Yunhao Zhou, Yi Wang, Lap-Pui ChauIEEE Trans. Multimedia 2021 · OpenReview

TL;DR

This paper introduces an "Attention-based Correlation Predictor" (ACP) for re-identification (re-ID) that mimics human memory to improve initial search results. By using a novel "Contextual Memory" and attention mechanisms, ACP effectively re-ranks images, significantly boosting performance on large-scale datasets.

What Problem Does This Paper Address?

Re-identification (re-ID) aims to find images of the same person or vehicle across different camera views. While deep learning has made great strides, initial search results (ranking lists) often contain errors due to variations in lighting, viewpoint, and occlusion. "Re-ranking" is a post-processing step that uses contextual information from neighboring images to refine these initial lists. Existing re-ranking methods often rely on hand-designed rules that struggle with complex relationships and large datasets, leading to performance degradation and high computational costs. This paper addresses the need for a more robust, efficient, and data-driven re-ranking approach.

Methods

The core of the proposed method is the Attention-based Correlation Predictor (ACP). It works in several stages:

  1. Multi-Block Feature Fusion: Instead of relying solely on high-level features from the deepest layers of a CNN backbone (like ResNet-50), ACP fuses features from multiple blocks. This allows it to capture richer, multi-scale information, including details from shallower layers.
  2. Context Aggregation via Transformer (BaseEncoder): The fused features from an image and its k-nearest neighbors are fed into a Transformer encoder. This module uses Multi-Head Attention (MHA) to aggregate contextual information, allowing each image's representation to be influenced by its neighbors.
  3. Contextual Memory Initialization: This is a key novel component. It mimics human memory by distilling probe-specific features from the aggregated context using attention. The memory cell stores multi-view descriptions of the probe image.
  4. Memory Refinement: To combat "feature pollution" from incorrectly retrieved neighbors, a refinement step further updates the memory cell using high-confidence samples from a "refinement sequence." This ensures the memory remains focused on discriminative features.
  5. Feature Reconstruction and Correlation Prediction: The refined Contextual Memory is used to reconstruct neighbor embeddings. The divergence between reconstructed and original embeddings is then used by a binary classifier to predict the correlation (i.e., the likelihood of a true match). These predicted correlations act as weights to move embeddings closer to their identity centers, improving cluster compactness.

Datasets

The proposed method was evaluated on six widely-used re-ID datasets:

  • Person Re-ID: Market1501, DukeMTMC-ReID, MSMT17
  • Vehicle Re-ID: VERI-Wild, VehicleID, VeRi-776

The paper highlights MSMT17 and VERI-Wild as being particularly challenging due to their large scale and real-world conditions.

Key Insights and Ideas

  • Mimicking Human Memory: The core idea of using a "Contextual Memory" to store and refine probe-specific information is inspired by how humans recall and verify information, leading to more robust re-ranking.
  • Attention for Contextualization: Attention mechanisms are crucial for both aggregating contextual information from neighbors and for distilling relevant features into the memory cell.
  • Feature Reconstruction for Discrimination: Reconstructing neighbor features using the memory and then analyzing the reconstruction error provides a powerful signal for predicting correlations, effectively separating true matches from false ones.
  • Moving Towards Identity Centers: The predicted correlations are used as weights to pull embeddings towards their identity centers, directly improving cluster compactness and thus retrieval accuracy.
  • Multi-Block Feature Fusion: Leveraging features from shallower CNN layers, in addition to deeper ones, provides richer representations that benefit re-ranking.

Key Results

  • State-of-the-Art Performance: ACP significantly outperforms existing re-ranking methods on large-scale datasets like VERI-Wild, MSMT17, and VehicleID, achieving an average of 4.83% CMC@1 and 14.83% mAP improvement.
  • Efficiency: The proposed method is significantly faster than some competing methods, e.g., taking only 3.6s for MSMT17 compared to 430.6s for k-reciprocal re-ranking.
  • Robustness: ACP shows strong performance across different dataset sizes and complexities, demonstrating robustness to varying neighborhood sizes.
  • Ablation Studies: Experiments confirm the effectiveness of each proposed module: Multi-Block Feature Fusion, BaseEncoder, Contextual Memory, and Memory Refinement.

Limitations and Future Work

  • Overfitting on Small Datasets: The paper notes that on very small datasets like Market1501 and DukeMTMC, where the baseline re-ID model already achieves near-perfect performance, ACP's ability to learn and improve is limited due to the lack of challenging cases. This suggests that ACP is most beneficial when there's a clear need for refinement.
  • Parameter Sensitivity: While the paper studies parameter sensitivity (e.g., k1, k2, number of heads, memory size), finding optimal parameters might still require careful tuning for specific applications.
  • Future Work: The authors suggest that their approach can be extended to other retrieval tasks beyond re-ID. Further exploration of memory architectures and attention mechanisms could also yield improvements.

Summary updated

Abstract

Re-ranking utilizes contextual information to optimize the initial ranking list of person or vehicle re-identification (re-ID), which boosts the retrieval performance at post-processing steps. This paper proposes a re-ranking network to predict the correlations between the probe and top-ranked neighbor samples. Specifically, all the feature embeddings of query and gallery images are expanded and enhanced by a linear combination of their neighbors, with the correlation prediction serving as discriminative combination weights. The combination process is equivalent to moving independent embeddings toward the identity centers, improving cluster compactness. For correlation prediction, we first aggregate the contextual information for probe's k-nearest neighbors via the Transformer encoder. Then, we distill and refine the probe-related features into the Contextual Memory cell via attention mechanism. Like humans that retrieve images by not only considering probe images but also memorizing the retrieved ones, the Contextual Memory produces multi-view descriptions for each instance. Finally, the neighbors are reconstructed with features fetched from the Contextual Memory, and a binary classifier predicts their correlations with the probe. Experiments on six widely-used person and vehicle re-ID benchmarks demonstrate the effectiveness of the proposed method. Especially, our method surpasses the state-of-the-art re-ranking approaches on large-scale datasets by a significant margin, i.e., with an average 4.83% CMC@1 and 14.83% mAP improvements on VERI-Wild, MSMT17, and VehicleID datasets.
BibTeX
@article{zhou2021moving,
  author = {Yunhao Zhou and Yi Wang and Lap-Pui Chau},
  title = {{Moving Towards Centers: Re-ranking with Attention and Memory for Re-identification}},
  journal = {IEEE Trans. Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=E9y47vcv_ue},
}
Figure