Abstract
Duplicate removal is a critical step to accomplish a reasonable amount of predictions in prevalent proposal-based object detection frameworks. Albeit simple and effective, most previous algorithms utilize a greedy process without making sufficient use of properties of input data. In this work, we design a new two-stage framework to effectively select the appropriate proposal candidate for each object. The first stage suppresses most of easy negative object proposals, while the second stage selects true positives in the reduced proposal set. These two stages share the same network structure, i.e., an encoder and a decoder formed as recurrent neural networks (RNN) with global attention and context gate. The encoder scans proposal candidates in a sequential manner to capture the global context information, which is then fed to the decoder to extract optimal proposals. In our extensive experiments, the proposed method outperforms other alternatives by a large margin.
Introduction
In modern object detection, frameworks typically generate thousands of candidate bounding boxes, or "proposals," for objects within an image. A significant challenge arises because many of these proposals overlap on the same object. The traditional solution is Non-Maximum Suppression (NMS), a greedy algorithm that retains the box with the highest confidence score and discards others with significant overlap.
However, NMS is fundamentally limited. It operates greedily, ignoring the global context and relationships between different proposals. Furthermore, it relies heavily on classification scores, which may not accurately reflect the quality of a box's localization. This paper addresses the "Oracle gap"—the performance difference between standard NMS and a theoretical perfect selection—by introducing a context-aware, two-stage RNN-based framework for duplicate removal.
Methodology
The proposed framework treats duplicate removal as a sequence-to-sequence problem, utilizing a two-stage architecture to refine object proposals.
Feature Embedding
For each proposal, the model extracts a comprehensive feature set:
- Appearance ($f_A$): High-dimensional features from the backbone detection network.
- Geometry ($f_G$): Bounding box coordinates and dimensions.
- Score ($f_S$): Classification confidence.
- Rank ($f_R$): Positional encoding based on the proposal's score rank.
Two-Stage Architecture
The system operates in two distinct phases:
- Stage I (Coarse Filter): Suppresses easy negative proposals (background noise).
- Stage II (Fine Filter): Selects the single best true positive from the remaining candidates.
Both stages utilize a Gated Recurrent Unit (GRU) encoder-decoder. The encoder scans proposals sequentially to capture global context, while the decoder re-scans them to make final "keep or kill" decisions. This process is enhanced by Global Attention, allowing the decoder to consider all proposals simultaneously, and a Context Gate that dynamically balances geometric and contextual features.
Experiments and Results
The model was evaluated on the MS COCO dataset, serving as a plug-in replacement for NMS in architectures like FPN, Mask R-CNN, and PANet.
Performance Gains
The proposed method consistently outperformed NMS and its variants (Soft-NMS, Box Voting). On the COCO validation set, it improved FPN's mAP from 37.1 to 38.6 and Mask R-CNN's from 38.9 to 40.6. Notably, the gains were most significant at higher IoU thresholds (mAP$_{75}$), indicating superior localization accuracy.
Efficiency and Insights
Despite the sequential nature of RNNs, the computational overhead is minimal, adding only 0.45 million parameters and approximately 19ms to inference time per image. Key insights include:
- Oracle Potential: Perfect proposal selection could boost COCO mAP by over 10 points, highlighting duplicate removal as a major bottleneck.
- Sequential Processing: Sorting proposals by score allows the RNN to learn the distribution of candidates and recognize duplicates based on previously seen high-quality boxes.
- Rank as a Feature: Positional encoding of rank provides a critical signal missing from raw scores.
Conclusion
This work demonstrates that replacing greedy NMS with a context-aware sequential model significantly improves object detection performance. By leveraging global relationships and a two-stage refinement process, the framework achieves state-of-the-art results with minimal overhead. Future work includes exploring end-to-end joint training with the detection network and accounting for inter-class relationships.
Summary updated