Learning to Rank Proposals for Object Detection

Zhiyu Tan, Xuecheng Nie, Qi Qian, Nan Li, Hao LiICCV 2019 · OpenReview

Abstract: Non-Maximum Suppression (NMS) is an essential step of modern object detection models for removing duplicated candidates. The efficacy of NMS heavily affects the final detection results. Prior works exploit suppression criterions relying on either the objectiveness derived from classification or the overlapness produced by regression, both of which are heuristically designed and fail to explicitly link with the suppression rank. To address this issue, in this paper, we propose a novel Learning-to-Rank (LTR) model to produce the suppression rank via a learning procedure, thus facilitating the candidate generation and lifting the detection performance. In particular, we define a ranking score based on IoU to indicate the ranks of candidates during the NMS step, where candidates with high ranking score will be reserved and the ones with low ranking score will be eliminated. We design a lightweight network to predict the ranking score. We introduce a ranking loss to supervise the generation of these ranking scores, which encourages candidates with IoU to the ground-truth to rank higher. To facilitate the training procedure, we design a novel sampling strategy via dividing candidates into different levels and select hard pairs to adopt in the training. During the inference phase, this module can be exploited as a plugin to the current object detector. The training and inference of the overall framework is end-to-end. Comprehensive experiments on benchmarks PASCAL VOC and MS COCO demonstrate the generality and effectiveness of our model for facilitating existing object detectors to state-of-the-art accuracy.

Introduction

Modern object detection frameworks typically generate a vast number of candidate bounding boxes, many of which are redundant. To refine these candidates into a final set of detections, Non-Maximum Suppression (NMS) is employed. However, the effectiveness of NMS is limited by its selection criterion. Traditionally, NMS relies on classification scores—the model's confidence that an object belongs to a specific class. This creates a "misalignment" problem: a proposal might have a high classification score but poor spatial alignment (low Intersection-over-Union, or IoU) with the ground truth. Conversely, a perfectly localized box might be discarded if its classification score is slightly lower. This paper addresses this issue by introducing a Learning-to-Rank (LTR) approach to explicitly predict the optimal suppression order.

Methods: The Rank-NMS Subnetwork

The authors propose a lightweight Rank-NMS subnetwork designed to be integrated into existing two-stage detectors, such as Faster R-CNN.

The Rank-NMS subnetwork architecture integrated into a standard detection pipeline.

As shown in the architecture above, the subnetwork consists of two fully connected layers that process RoI-aligned features to output a dedicated ranking score. Unlike standard regression, this module is trained using a margin-based ranking loss. This loss function ensures that boxes with higher IoU relative to the ground truth are assigned higher ranking scores than those with lower IoU. To optimize training, a Hard Pair Sampling strategy is used, focusing the model on ambiguous pairs of candidates that are difficult to rank correctly. During inference, the final score for NMS is a weighted fusion of this learned ranking score and the traditional classification score, balancing category confidence with localization precision.

Experimental Results

The LTR model was evaluated on the PASCAL VOC and MS COCO datasets. The results demonstrate that decoupling classification from localization significantly improves detection performance.

Recall vs. Matching IoU curves comparing standard NMS, Soft-NMS, and the proposed Rank-NMS.

Key findings include:

  • Performance Gains: On PASCAL VOC, integrating the LTR module into Faster R-CNN (ResNet-50) improved Average Precision (AP) from 54.16% to 58.75%. On MS COCO, a Cascade R-CNN (ResNet-101) baseline reached a state-of-the-art 43.7% AP.
  • High-Precision Localization: As illustrated in the recall curves, the improvements are most pronounced at high IoU thresholds (e.g., $AP^{90}$). This confirms that Rank-NMS is far more effective at preserving the most accurately aligned bounding boxes compared to traditional NMS or Soft-NMS.
  • Efficiency: The module is highly efficient; the inference overhead is minimal, with only a slight drop in frames per second (e.g., from 9.3 to 8.3 fps on a Titan P100), making it suitable for practical deployment.

Conclusion and Future Work

By replacing heuristic suppression rules with a learned ranking mechanism, this work successfully mitigates the misalignment between classification confidence and localization accuracy. While currently optimized for two-stage detectors, the Rank-NMS module provides a general and effective "plug-and-play" solution for enhancing object detection precision. Future research may explore extending this ranking logic to one-stage detectors and further automating the hyperparameter tuning for score fusion.

Summary updated

Abstract

Non-Maximum Suppression (NMS) is an essential step of modern object detection models for removing duplicated candidates. The efficacy of NMS heavily affects the final detection results. Prior works exploit suppression criterions relying on either the objectiveness derived from classification or the overlapness produced by regression, both of which are heuristically designed and fail to explicitly link with the suppression rank. To address this issue, in this paper, we propose a novel Learning-to-Rank (LTR) model to produce the suppression rank via a learning procedure, thus facilitating the candidate generation and lifting the detection performance. In particular, we define a ranking score based on IoU to indicate the ranks of candidates during the NMS step, where candidates with high ranking score will be reserved and the ones with low ranking score will be eliminated. We design a lightweight network to predict the ranking score. We introduce a ranking loss to supervise the generation of these ranking scores, which encourages candidates with IoU to the ground-truth to rank higher. To facilitate the training procedure, we design a novel sampling strategy via dividing candidates into different levels and select hard pairs to adopt in the training. During the inference phase, this module can be exploited as a plugin to the current object detector. The training and inference of the overall framework is end-to-end. Comprehensive experiments on benchmarks PASCAL VOC and MS COCO demonstrate the generality and effectiveness of our model for facilitating existing object detectors to state-of-the-art accuracy.
BibTeX
@inproceedings{tan2019learning,
  author = {Zhiyu Tan and Xuecheng Nie and Qi Qian and Nan Li and Hao Li},
  title = {{Learning to Rank Proposals for Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=ASZwVQCMd3Uj},
}
Figure