TL;DR
This paper introduces Rank & Sort (RS) Loss, a novel loss function for object detection and instance segmentation that not only ranks positive detections above negatives but also sorts positives by their localization quality (IoU). This approach simplifies training by eliminating the need for auxiliary heads and sampling heuristics, while also improving robustness to class imbalance and consistently outperforming existing methods.
What Problem Does This Paper Address?
Training deep object detection and instance segmentation models often involves complex loss functions with numerous hyperparameters that require extensive tuning. Existing ranking-based losses, while improving robustness to class imbalance, typically focus only on ranking positives above negatives and don't explicitly model the quality of positive predictions relative to each other. This can lead to suboptimal performance and makes fair comparison between methods challenging. Furthermore, the non-differentiable nature of ranking and sorting operations poses a significant challenge for optimization.
Methods
The core innovation is the Rank & Sort (RS) Loss function. It addresses two key challenges:
- Ranking Positives Above Negatives: This is a standard approach in some ranking-based losses, ensuring that correct detections are prioritized over incorrect ones.
- Sorting Positives by Localization Quality: This is the novel aspect. RS Loss sorts positive detections based on their Intersection-over-Union (IoU) with ground truth boxes. This means that detections with better localization are ranked higher.
To handle the non-differentiable nature of ranking and sorting, the authors propose an "Identity Update" reformulation. This technique allows for error-driven updates to be incorporated into backpropagation, making the ranking and sorting operations optimizable. This approach also provides interpretable loss values and uniquely accounts for intra-class errors (i.e., errors among positive predictions).
RS Loss simplifies training by:
- Eliminating Auxiliary Heads: The sorting objective directly supervises the classification head, removing the need for separate heads for centerness or IoU prediction.
- Removing Sampling Heuristics: RS Loss is inherently robust to class imbalance, negating the need for complex sampling strategies.
- Tuning-Free Task Balancing: The authors use simple, tuning-free coefficients to balance different tasks within the detector, further reducing the hyperparameter burden.
Datasets
The paper uses the following standard datasets for training and evaluation:
- COCO: A widely used benchmark for object detection and instance segmentation.
- LVIS: A long-tailed dataset for instance segmentation, known for its significant class imbalance.
Key Insights and Ideas
- Sorting Positives is Crucial: The paper demonstrates that not only ranking positives above negatives but also sorting positives by their localization quality (IoU) leads to significant performance gains. This aligns better with how detectors are evaluated (e.g., by AP) and how non-maximum suppression (NMS) works.
- "Identity Update" for Differentiable Ranking: The proposed Identity Update is a clever way to make non-differentiable ranking and sorting operations trainable via backpropagation, offering interpretable loss values.
- Simplification Through Robustness: RS Loss's inherent robustness to class imbalance and its direct supervision of the classification head significantly reduce the complexity of training pipelines, eliminating the need for auxiliary heads and sampling heuristics.
- Tuning-Free is Achievable: The authors show that by carefully designing the loss and using simple task-balancing strategies, it's possible to achieve state-of-the-art results with minimal hyperparameter tuning (primarily just the learning rate).
Key Results
The RS Loss consistently outperforms baselines across various detectors and tasks:
- Object Detection:
- Improves Faster R-CNN by ~3 box AP on COCO.
- Outperforms aLRP Loss (a ranking-based baseline) by ~2 box AP on COCO.
- Achieves 47.8 AP on COCO test-dev with RS-R-CNN, outperforming similarly trained Faster R-CNN and Dynamic R-CNN.
- Instance Segmentation:
- Improves Mask R-CNN by 3.5 mask AP on the LVIS dataset (~7 AP for rare classes).
- Outperforms Mask-scoring R-CNN (which uses an additional mask IoU head) by 0.4 mask AP on COCO.
- Robustness: RS Loss demonstrates strong performance even with extreme class imbalance (e.g., 1:10470 pos:neg ratio on LVIS) without requiring any tuning.
- Efficiency: While RS Loss has a slightly higher training time (approx. 1.5x), it can lead to faster inference due to higher confidence scores, allowing for more aggressive score thresholding.
Limitations and Future Work
- Training Time: RS Loss has a quadratic time complexity, leading to approximately 1.5x longer training iterations compared to score-based losses.
- Inference Efficiency Tuning: While RS Loss can lead to faster inference, it requires careful tuning of the confidence score threshold for NMS.
- Future Work: The paper suggests exploring the generalizability of the "Identity Update" to other non-differentiable tasks and further investigating the impact of different continuous labels (beyond IoU and Dice) for supervising localization. The authors also highlight the potential for further improvements by combining RS Loss with other advanced techniques.
Summary updated