Memory-Based Neighbourhood Embedding for Visual Recognition

Suichan Li, Dapeng Chen, Bin Liu, Nenghai Yu, Rui Zhao

Abstract

Traditional visual recognition models typically generate image embeddings by processing single images in isolation, often failing to capture the rich contextual relationships present in similar instances. This paper introduces Memory-based Neighbourhood Embedding (MNE), a framework that augments standard convolutional neural networks (CNNs) with an episodic memory and a tree-graph structure. By leveraging supervised attention to aggregate information from a target image's neighborhood, MNE refines feature representations to better handle high intra-class variance and inter-class similarity. Experimental results demonstrate that MNE achieves state-of-the-art performance across person search and few-shot learning benchmarks.

Introduction

In visual recognition, the primary objective is to map images into a high-dimensional embedding space where semantic similarity corresponds to geometric proximity. Conventional methods often rely on deeper architectures or specialized loss functions to improve these embeddings. However, these approaches generally estimate features based on a single image, which can be problematic when intra-class variance is high (e.g., varying camera angles) or inter-class similarity is significant (e.g., different individuals in similar attire).

The authors propose that an image's identity is more accurately resolved when viewed within the context of its "neighborhood"—the set of visually similar images. The MNE framework addresses the challenge of efficiently identifying and aggregating these neighbors during training without introducing noise from irrelevant instances.

The MNE Framework

The MNE architecture consists of three core components designed to contextualize image features:

  1. Episodic Memory Module: An external memory stores features and labels for a large number of instances. This module is updated dynamically as the CNN backbone evolves, allowing the model to look beyond the current mini-batch to find relevant neighbors.
  2. Neighbourhood Tree-Graph: For a given target image, the model retrieves its $K$ nearest neighbors from memory to construct a tree of depth $H$. The target image serves as the root, with its neighbors as children and their respective neighbors as grandchildren, capturing the local data manifold.
  3. Aggregation with Supervised Attention (ASA): Information is propagated from the leaf nodes up to the root. Unlike standard pooling, MNE employs a supervised attention mechanism trained via binary cross-entropy. This ensures that higher weights are assigned to neighbors sharing the same class label as the parent node, making the final embedding robust to outliers.
Overview of the MNE framework, illustrating the flow from target image and episodic memory to neighborhood-based embedding refinement via tree-graph aggregation.

Experimental Evaluation

The MNE method was evaluated on two distinct tasks: Person Search (Re-Identification) and Few-Shot Learning.

  • Person Search: Utilizing the CUHK03 and DukeMTMC datasets, MNE addressed challenges like lighting variations and occlusions. On DukeMTMC, MNE achieved 87.5% mAP and 90.4% Rank-1 accuracy, significantly outperforming previous spectral-based methods.
  • Few-shot Learning: On miniImageNet and tieredImageNet, MNE demonstrated superior generalization. In the 1-shot 5-way task on miniImageNet, MNE reached 60.20% accuracy, surpassing established models like TPN and RelationNet.

Key Insights

  • Contextual Correction: Neighborhood analysis acts as a corrective force; if two images from different classes are initially close, their divergent neighborhoods help the model distinguish them.
  • Explicit Supervision: By explicitly supervising the attention mechanism to favor same-class neighbors, the model becomes significantly more robust than those using implicit attention.
  • Memory Efficiency: The episodic memory effectively breaks the batch-size barrier imposed by GPU limitations, allowing for large-scale instance interaction.
t-SNE visualization comparing baseline embeddings with those generated by MNE, demonstrating significantly improved class separation and tighter clusters.

Conclusion

Memory-based Neighbourhood Embedding represents a significant shift from isolated image processing to context-aware feature estimation. While the method introduces additional computational complexity during tree construction and aggregation, the performance gains in retrieval and few-shot scenarios are substantial. Future work may focus on reducing the latency of the iterative aggregation process for real-time applications and exploring the method's performance in extremely sparse data environments.

Summary updated

Abstract

Learning discriminative image feature embeddings is of great importance to visual recognition. To achieve better feature embeddings, most current methods focus on designing different network structures or loss functions, and the estimated feature embeddings are usually only related to the input images. In this paper, we propose Memory-based Neighbourhood Embedding (MNE) to enhance a general CNN feature by considering its neighbourhood. The method aims to solve two critical problems, i.e., how to acquire more relevant neighbours in the network training and how to aggregate the neighbourhood information for a more discriminative embedding. We first augment an episodic memory module into the network, which can provide more relevant neighbours for both training and testing. Then the neighbours are organized in a tree graph with the target instance as the root node. The neighbourhood information is gradually aggregated to the root node in a bottom-up manner, and aggregation weights are supervised by the class relationships between the nodes. We apply MNE on image search and few shot learning tasks. Extensive ablation studies demonstrate the effectiveness of each component, and our method significantly outperforms the state-of-the-art approaches.
BibTeX
@inproceedings{li2019memory,
  author = {Suichan Li and Dapeng Chen and Bin Liu and Nenghai Yu and Rui Zhao},
  title = {{Memory-Based Neighbourhood Embedding for Visual Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00620},
}
Figure