RankDetNet: Delving Into Ranking Constraints for Object Detection

Ji Liu, Dong Li, Rongzhang Zheng, Lu Tian, Yi ShanCVPR 2021 · OpenReview

Abstract

Modern object detection approaches cast detecting objects as optimizing two subtasks of classification and localization simultaneously. Existing methods often learn the classification task by optimizing each proposal separately and neglect the relationship among different proposals. Such detection paradigm also encounters the mismatch between classification and localization due to the inherent discrepancy of their optimization targets. In this work, we propose a ranking-based optimization algorithm for harmoniously learning to rank and localize proposals in lieu of the classification task. To this end, we comprehensively investigate three types of ranking constraints, i.e., global ranking, class-specific ranking and IoU-guided ranking losses. The global ranking loss encourages foreground samples to rank higher than background. The class-specific ranking loss ensures that positive samples rank higher than negative ones for each specific class. The IoU-guided ranking loss aims to align each pair of confidence scores with the associated pair of IoU overlap between two positive samples of a specific class. Our ranking constraints can sufficiently explore the relationships between samples from three different perspectives. They are easy-to-implement, compatible with mainstream detection frameworks and computation-free for inference. Experiments demonstrate that our RankDetNet consistently surpasses prior anchor-based and anchor-free baselines, e.g., improving RetinaNet baseline by 2.5% AP on the COCO test-dev set without bells and whistles. We also apply the proposed ranking constraints for 3D object detection and achieve improved performance, which further validates the superiority and generality of our method.

Introduction

Traditional object detection models typically optimize two distinct subtasks: classification and localization. Most current paradigms treat classification as a "point-wise" problem, where each proposal is evaluated in isolation using losses like Cross-Entropy or Focal Loss. This approach has two significant drawbacks. First, it ignores the relative relationships between proposals; in detection, the relative ranking of boxes is often more critical than their absolute probability scores. Second, there is a persistent mismatch between classification confidence and localization accuracy. During Non-Maximum Suppression (NMS), a detector might retain a box with high confidence but poor localization while discarding a better-aligned box with a slightly lower score.

Methodology: Ranking-Based Optimization

RankDetNet addresses these issues by reframing detection as a ranking problem. Instead of standard point-wise classification, it introduces three pair-wise ranking constraints that explore sample relationships from different perspectives.

Conceptual comparison between standard classification and the proposed ranking constraints.

Ranking Constraints

  1. Global Ranking Loss: This constraint encourages all foreground samples to rank higher than background samples across the entire image, regardless of their specific class. It employs a dynamic re-weighting factor to prioritize "reverse pairs" where background scores currently exceed foreground scores.
  2. Class-Specific Ranking Loss: This ensures that for any given category, positive samples are ranked higher than negative ones (including background and other object classes).
  3. IoU-Guided Ranking Loss: This is the primary mechanism for alignment. For two positive samples of the same class, the sample with the higher Intersection-over-Union (IoU) with the ground truth is forced to have a higher confidence score. This directly couples confidence with localization quality.

Network Architecture

The RankDetNet architecture maintains the standard Feature Pyramid Network (FPN) and detection head structure. The ranking constraints are integrated into the training objective, replacing or augmenting the traditional classification loss.

Architectural comparison between a baseline detection network and the RankDetNet framework.

To ensure training stability, a "warm-up" strategy is employed where the model is initially trained with standard Focal Loss before transitioning to the ranking-based objectives.

Experimental Evaluation

The proposed method was evaluated on the MS COCO 2017 and KITTI datasets. RankDetNet consistently outperformed strong baselines across various backbones.

On the COCO test-dev set, RankDetNet improved the RetinaNet baseline by 2.5% AP and the FCOS-ATSS baseline by 2.4% AP. Notably, the performance gains were more significant at higher IoU thresholds (e.g., $AP_{75}$), indicating that the ranking constraints effectively improve localization precision. For 3D object detection on the KITTI benchmark, applying these constraints to the SA-SSD baseline resulted in a 2.02% AP improvement for the "Car" category.

Conclusion

RankDetNet demonstrates that reframing object detection as a ranking problem significantly improves performance by better utilizing sample relationships and aligning confidence with localization. Because these constraints are only applied during training, the model achieves these gains with zero additional computational cost during inference. This approach offers a general and effective optimization strategy compatible with both anchor-based and anchor-free detection frameworks.

Summary updated

Abstract

Modern object detection approaches cast detecting objects as optimizing two subtasks of classification and localization simultaneously. Existing methods often learn the classification task by optimizing each proposal separately and neglect the relationship among different proposals. Such detection paradigm also encounters the mismatch between classification and localization due to the inherent discrepancy of their optimization targets. In this work, we propose a ranking-based optimization algorithm for harmoniously learning to rank and localize proposals in lieu of the classification task. To this end, we comprehensively investigate three types of ranking constraints, i.e., global ranking, class-specific ranking and IoU-guided ranking losses. The global ranking loss encourages foreground samples to rank higher than background. The class-specific ranking loss ensures that positive samples rank higher than negative ones for each specific class. The IoU-guided ranking loss aims to align each pair of confidence scores with the associated pair of IoU overlap between two positive samples of a specific class. Our ranking constraints can sufficiently explore the relationships between samples from three different perspectives. They are easy-to-implement, compatible with mainstream detection frameworks and computation-free for inference. Experiments demonstrate that our RankDetNet consistently surpasses prior anchor-based and anchor-free baselines, e.g., improving RetinaNet baseline by 2.5% AP on the COCO test-dev set without bells and whistles. We also apply the proposed ranking constraints for 3D object detection and achieve improved performance, which further validates the superiority and generality of our method.
BibTeX
@inproceedings{liu2021rankdetnet,
  author = {Ji Liu and Dong Li and Rongzhang Zheng and Lu Tian and Yi Shan},
  title = {{RankDetNet: Delving Into Ranking Constraints for Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=d4oHsuB5IY},
}
Figure