Re-Ranking via Metric Fusion for Object Retrieval and Person Re-Identification

Song Bai, Peng Tang, Philip H. S. Torr, Longin Jan LateckiCVPR 2019 · OpenReview

Abstract

This work studies the unsupervised re-ranking procedure for object retrieval and person re-identification with a specific concentration on an ensemble of multiple metrics (or similarities). While the re-ranking step is involved by running a diffusion process on the underlying data manifolds, the fusion step can leverage the complementarity of multiple metrics. We give a comprehensive summary of existing fusion with diffusion strategies, and systematically analyze their pros and cons. Based on the analysis, we propose a unified yet robust algorithm which inherits their advantages and discards their disadvantages. Hence, we call it Unified Ensemble Diffusion (UED). More interestingly, we derive that the inherited properties indeed stem from a theoretical framework, where the relevant works can be elegantly summarized as special cases of UED by imposing additional constraints on the objective function and varying the solver of similarity propagation. Extensive experiments with 3D shape retrieval, image retrieval and person re-identification demonstrate that the proposed framework outperforms the state of the arts, and at the same time suggest that re-ranking via metric fusion is a promising tool to further improve the retrieval performance of existing algorithms.

Introduction

In visual retrieval—whether searching for a specific 3D CAD model or a person across multiple security cameras—the goal is to find the most similar matches to a query. Traditional systems rely on "pairwise" comparisons, which calculate similarity between a query and database items in isolation. However, this ignores the global "context"—how items relate to each other within the database manifold.

Re-ranking via diffusion processes addresses this by considering the global structure of the data. Meanwhile, Metric Fusion attempts to combine different "views" of the data (e.g., color, shape, or deep features) to improve results. Existing methods for combining these concepts often face trade-offs: Naive Fusion (NF) is fast but easily poisoned by noisy metrics; Tensor Product Fusion (TPF) is robust but limited to two metrics; and Regularized Ensemble Diffusion (RED) is robust but computationally expensive. This paper introduces Unified Ensemble Diffusion (UED) to overcome these limitations.

Methods

UED alternates between two primary steps to achieve robust and efficient re-ranking:

  1. The Diffusion Step: UED computes a weighted average of all input similarity matrices and runs a single diffusion process. This maintains the speed of Naive Fusion while allowing for complex interplay between different metrics.
  2. The Weight Learning Step: The framework treats metric weights as variables to be optimized. By introducing the replicator equation (from evolutionary game theory) to the re-ranking stage, the system automatically "down-weights" noisy or unhelpful metrics.

Theoretical analysis proves that NF, TPF, and RED are special cases of UED under specific constraints. By relaxing these constraints, UED inherits the speed of NF, the robustness of RED, and the metric-interplay benefits of TPF.

Experimental Evaluation

The authors evaluated UED on three distinct retrieval challenges: 3D Shape Retrieval (ModelNet40/10), Image Retrieval (Holidays, Ukbench), and Person Re-Identification (Market-1501).

Visual examples of the person re-identification task, showing probe images and corresponding gallery matches from the Market-1501 dataset.

Key Results

  • 3D Shape Retrieval: UED achieved state-of-the-art performance on ModelNet benchmarks. On ModelNet40, it reached an AUC of 88.05%, significantly outperforming RED (87.03%) and Naive Fusion (85.26%).
  • Person Re-ID: On the Market-1501 dataset, UED achieved a mean Average Precision (mAP) of 92.75%, marking one of the first works to surpass the 90% threshold on this benchmark.
  • Noise Robustness: In experiments involving a deliberately noisy metric (B4), UED maintained high accuracy by assigning it a weight near zero (0.014), whereas Naive Fusion's performance dropped significantly.
Comparison of UED with state-of-the-art methods on ModelNet40 and ModelNet10 datasets. UED (ours) achieves the best performance (red) or second-best (blue) across metrics.

Conclusion

Unified Ensemble Diffusion provides a robust, efficient, and theoretically grounded approach to metric fusion. By leveraging the interplay between complementary metrics and automatically suppressing noise, UED sets new benchmarks in visual retrieval. Future work will explore integrating this contextual manifold information directly into the training phase of deep models for end-to-end re-ranking.

Summary updated

Abstract

This work studies the unsupervised re-ranking procedure for object retrieval and person re-identification with a specific concentration on an ensemble of multiple metrics (or similarities). While the re-ranking step is involved by running a diffusion process on the underlying data manifolds, the fusion step can leverage the complementarity of multiple metrics. We give a comprehensive summary of existing fusion with diffusion strategies, and systematically analyze their pros and cons. Based on the analysis, we propose a unified yet robust algorithm which inherits their advantages and discards their disadvantages. Hence, we call it Unified Ensemble Diffusion (UED). More interestingly, we derive that the inherited properties indeed stem from a theoretical framework, where the relevant works can be elegantly summarized as special cases of UED by imposing additional constraints on the objective function and varying the solver of similarity propagation. Extensive experiments with 3D shape retrieval, image retrieval and person re-identification demonstrate that the proposed framework outperforms the state of the arts, and at the same time suggest that re-ranking via metric fusion is a promising tool to further improve the retrieval performance of existing algorithms.
BibTeX
@inproceedings{bai2019re,
  author = {Song Bai and Peng Tang and Philip H. S. Torr and Longin Jan Latecki},
  title = {{Re-Ranking via Metric Fusion for Object Retrieval and Person Re-Identification}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=DN3mcNI5P3},
}

Loading related content...

Figure