PyRetri: A PyTorch-based Library for Unsupervised Image Retrieval by Deep Convolutional Neural Networks

Benyi Hu, Ren-Jie Song, Xiu-Shen Wei, Yazhou Yao, Xian-Sheng Hua, Yuehu Liu

Introduction

Content-Based Image Retrieval (CBIR) has seen significant advancements with the advent of Deep Learning. However, the field often struggles with an "implementation mess"—a lack of standardized software that covers the entire retrieval pipeline from preprocessing to reranking. This fragmentation makes it difficult to reproduce results or isolate the impact of specific components like pooling methods or post-processing algorithms. PyRetri addresses this by providing a unified, modular PyTorch-based library designed to standardize unsupervised image retrieval.

The PyRetri Framework

PyRetri introduces a modular architecture that decomposes the retrieval process into three core stages: Feature Extraction, Indexing, and Evaluation.

The modular pipeline of PyRetri, encompassing feature extraction, indexing, and evaluation.
  1. Feature Extraction Module: This module handles the forward pass of the network. It supports various backbones (e.g., VGG, ResNet) and uses a "hooking" mechanism to extract features from any layer. It includes diverse aggregation methods such as GeM, R-MAC, and SCDA to convert convolutional maps into compact descriptors.
  2. Indexing Module: Serving as the engine of the process, this module manages post-processing (SVD, PCA), feature enhancement (Database Augmentation, Query Expansion), and distance metrics. It also implements advanced reranking techniques like k-reciprocal encoding.
  3. Evaluation Module: This provides standardized metrics (mAP, Recall) and visualization tools for failure analysis.

A key innovation is the Configuration Search Tool, which leverages human-readable YACS config files to allow researchers to exhaustively search for the optimal combination of components for a given dataset.

Experimental Evaluation

The authors validated PyRetri across several benchmarks, including Oxford5k (landmarks), CUB-200-2011 (fine-grained birds), Indoor (scenes), and Caltech101 (generic objects). The results demonstrate that finding the "sweet spot" in configuration can lead to massive accuracy boosts without any additional training.

Performance comparison between baseline and optimal configurations across multiple datasets.

As shown in the table above, the Indoor dataset saw an mAP jump from a baseline of 23.9 to 63.7 (+39.8). Similarly, Oxford5k improved from 46.3 to 72.9 (+26.6). These gains highlight that sophisticated unsupervised pipelines using pre-trained ImageNet models can achieve high accuracy if the post-processing and indexing stages are optimized.

Furthermore, PyRetri proved effective in specialized domains like Person Re-Identification (Re-ID), matching or exceeding specialized implementations on the Market-1501 dataset with an mAP of 84.8. Efficiency remains a priority, with average inference times staying below 8ms per image.

Conclusion

PyRetri standardizes the deep image retrieval pipeline, proving that modularity and configuration optimization are as critical as model architecture. By providing a unified framework, it enables fair comparison and rapid prototyping in the CBIR community. Future work may involve integrating more supervised metric learning components to further enhance its utility.

Availability: The project is open-source and available at https://github.com/PyRetri/PyRetri/.

Summary updated

BibTeX
@inproceedings{hu2020pyretri,
  author = {Benyi Hu and Ren-Jie Song and Xiu-Shen Wei and Yazhou Yao and Xian-Sheng Hua and Yuehu Liu},
  title = {{PyRetri: A PyTorch-based Library for Unsupervised Image Retrieval by Deep Convolutional Neural Networks}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=HUjaPfbVwyP},
}
Figure