A Ranking-based, Balanced Loss Function Unifying Classification and Localisation in Object Detection

Kemal Oksuz, Baris Can Cam, Emre Akbas, Sinan Kalkan

Abstract

We propose average Localisation-Recall-Precision (aLRP), a unified, bounded, balanced and ranking-based loss function for both classification and localisation tasks in object detection. aLRP extends the Localisation-Recall-Precision (LRP) performance metric (Oksuz et al., 2018) inspired from how Average Precision (AP) Loss extends precision to a ranking-based loss function for classification (Chen et al., 2020). aLRP has the following distinct advantages: (i) aLRP is the first ranking-based loss function for both classification and localisation tasks. (ii) Thanks to using ranking for both tasks, aLRP naturally enforces high-quality localisation for high-precision classification. (iii) aLRP provides provable balance between positives and negatives. (iv) Compared to on average $\sim$6 hyperparameters in the loss functions of state-of-the-art detectors, aLRP Loss has only one hyperparameter, which we did not tune in practice. On the COCO dataset, aLRP Loss improves its ranking-based predecessor, AP Loss, up to around $5$ AP points, achieves $48.9$ AP without test time augmentation and outperforms all one-stage detectors.

Introduction

Modern object detection involves two primary tasks: identifying the object class (classification) and regressing bounding box coordinates (localization). Traditionally, these tasks are optimized using a weighted sum of separate loss functions: $\mathcal{L} = \mathcal{L}_c + w_r\mathcal{L}_r$. This approach suffers from several critical issues. First, there is a "task disconnect" where high classification scores do not necessarily correlate with high-quality localization. Second, the balancing weight $w_r$ is a sensitive hyperparameter that is difficult and expensive to tune. Finally, the massive class imbalance between background and foreground samples requires complex sampling strategies or specialized losses like Focal Loss to prevent background samples from dominating the training signal.

The aLRP Loss Framework

To address these challenges, we introduce the average Localisation-Recall-Precision (aLRP) Loss. This unified, ranking-based loss function extends the LRP performance metric into a differentiable training objective.

Ranking-based Optimization

Unlike traditional losses that treat classification and localization independently, aLRP treats both as a ranking problem. Since ranking functions are non-differentiable step functions, we utilize a generalized error-driven optimization framework. This allows for the calculation of gradients by defining primary terms and target values for the non-differentiable components.

Gradients for positives, negatives, and box parameters in the aLRP framework.

As shown in the figure above, the framework defines specific gradient updates for positive examples ($p_i$), negative examples ($n_j$), and the box parameters ($B$). This unified formulation ensures that for a detection to achieve a high rank (and low loss), it must possess both a high classification score and a high Intersection-over-Union (IoU) with the ground truth.

Self-Balancing (SB) Mechanism

A key contribution is the introduction of a dynamic self-balancing mechanism. To eliminate the need for manual tuning of the localization weight $w_r$, we use the ratio of classification to localization gradients from the previous epoch to scale current updates. This ensures that both tasks contribute effectively to the learning process throughout training.

Experimental Results

We evaluated aLRP Loss on the industry-standard COCO dataset using various architectures, including RetinaNet, FoveaBox, and Faster R-CNN.

Performance and Efficiency

The aLRP Loss significantly outperformed its predecessor, AP Loss, by up to 5 AP points. Our best model, utilizing a ResNeXt-101 backbone, achieved 48.9 AP on COCO test-dev, surpassing all existing one-stage detectors at the time. Furthermore, when combined with Adaptive Training Sample Selection (ATSS), aLRP improved training efficiency by 34% by reducing the required number of anchors without sacrificing accuracy.

Evolution of aLRP loss components and the self-balancing ratio during training.

The figure above illustrates the stability of the training process under the self-balancing mechanism. Despite the inherent complexity of ranking-based losses, the components converge smoothly. The self-balancing feature proved robust even when initialized with wildly varying weights, consistently leading to nearly identical final performance.

Conclusion

The aLRP Loss provides a theoretically grounded and practically effective solution for unifying classification and localization in object detection. By leveraging a ranking-based approach, it naturally balances positive and negative samples and enforces a strong correlation between confidence scores and bounding box quality. With significantly fewer hyperparameters than state-of-the-art alternatives, aLRP simplifies the training process while achieving superior results.

Summary updated

Abstract

We propose average Localisation-Recall-Precision (aLRP), a unified, bounded, balanced and ranking-based loss function for both classification and localisation tasks in object detection. aLRP extends the Localisation-Recall-Precision (LRP) performance metric (Oksuz et al., 2018) inspired from how Average Precision (AP) Loss extends precision to a ranking-based loss function for classification (Chen et al., 2020). aLRP has the following distinct advantages: (i) aLRP is the first ranking-based loss function for both classification and localisation tasks. (ii) Thanks to using ranking for both tasks, aLRP naturally enforces high-quality localisation for high-precision classification. (iii) aLRP provides provable balance between positives and negatives. (iv) Compared to on average $\sim$6 hyperparameters in the loss functions of state-of-the-art detectors, aLRP Loss has only one hyperparameter, which we did not tune in practice. On the COCO dataset, aLRP Loss improves its ranking-based predecessor, AP Loss, up to around $5$ AP points, achieves $48.9$ AP without test time augmentation and outperforms all one-stage detectors. Code available at: https://github.com/kemaloksuz/aLRPLoss .
BibTeX
@inproceedings{oksuz2020ranking,
  author = {Kemal Oksuz and Baris Can Cam and Emre Akbas and Sinan Kalkan},
  title = {{A Ranking-based, Balanced Loss Function Unifying Classification and Localisation in Object Detection}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=9IdoCwgOwx},
}
Figure