TL;DR
Supervised post-processing can significantly boost the performance of "frozen" object detectors (where weights cannot be changed) by re-evaluating discarded hypotheses. This paper demonstrates that while richer feature sets like "Full100" improve detection, the real-world cost depends on implementation optimization rather than feature dimensionality, and improved performance often involves a complex "churn" of gaining new objects while losing some previously correct ones.
What Problem Does This Paper Address?
Standard object detectors (like Faster R-CNN) generate thousands of candidate boxes (hypotheses), but most are thrown away during a step called Non-Maximum Suppression (NMS) to prevent overlapping duplicates. Some of these discarded boxes actually contain real objects that the detector "saw" but failed to prioritize.
In many practical scenarios, you can't "fine-tune" the detector—perhaps you're accessing it via a fixed API or using a specialized hardware accelerator. This is a frozen detector. The authors ask: If we can't change the detector, how much can we improve its output just by smarter post-processing? Specifically, they investigate which pieces of information (scores, box geometry, or context) are worth the computational cost of processing.
Methods
The authors propose a framework for Supervised Post-Processing that treats the detector's internal outputs as features for a secondary, lightweight classifier (a Histogram Gradient-Boosted tree, or HGB).
Key components include:
- The Full100 Readout: A 100-dimensional feature vector extracted for every candidate box. It includes 91 class probabilities, 6 geometric terms (coordinates, area, aspect ratio), and 3 "native-set summaries" that describe how a candidate relates to the detector's default, high-confidence outputs.
- Comparison Models: They compare Full100 against simpler baselines (like using only person/background scores) and established methods like MetaDetect, which uses neighborhood statistics (how many other boxes are nearby).
- Execution Optimization: To ensure a fair "complete cost" comparison, they optimized the neighborhood statistics calculations (C99 implementation) to see if complex features are inherently slower.
Datasets
The study uses several challenging benchmarks to test the limits of person detection:
- CrowdHuman: The primary dataset, known for extreme overlapping and dense crowds.
- MOT17Det: Used to test "source-frozen transfer"—seeing if a post-processor trained on CrowdHuman works on video frames without further tuning.
- MIAP (More Inclusive People Annotations): A subset of Open Images used to test how the models handle different demographic distributions.
Key Insights and Ideas
- Context is King: The most significant performance gains didn't come from having more class probabilities, but from "native-set context"—knowing how a candidate box overlaps with the boxes the detector already decided were good.
- The "Churn" of Discoveries: A higher True Positive (TP) count doesn't mean the new model is a strict upgrade. As shown in the figure below, Full100 gains many new objects but also "loses" some that the raw detector originally caught. This "replacement" behavior is critical for engineers to understand before deploying a new policy.
- Dimension $\neq$ Cost: A common assumption is that more features = slower code. The authors found that after optimization, the 141-dimensional MetaStats model was nearly as fast as the 100-dimensional Full100 model (~95ms). The bottleneck is often data transfer and setup, not the number of features.
- Feature Non-Monotonicity: Simply adding more features doesn't always help. For instance, using all 91 class slots sometimes performed worse than just using the person/background scores unless geometry and context were also included.
Key Results
- Detection Boost: On CrowdHuman, the Full100 model matched 6,004 objects, a significant jump over the optimized raw baseline (5,848 objects).
- Efficiency: The "complete warm cost" (the time from reading an image to final output) for Full100 was approximately 95 ms on an NVIDIA A800 GPU. This represents only a ~6-7% overhead compared to raw processing, making it a viable "drop-in" improvement.
- Transferability: While the model trained on CrowdHuman transferred well to MOT17Det (showing a gain in True Positives), it showed no improvement on MIAP, suggesting that post-processing is sensitive to the specific "look" and box geometry of the training data.
Limitations and Future Work
- Task Specificity: The study focuses exclusively on person detection. Results might differ for multi-class datasets with very different object scales (e.g., COCO).
- Hardware Dependency: Latency measurements are specific to the tested hardware (NVIDIA A800). Performance on edge devices might favor simpler feature sets.
- Label-Free Adaptation: The current method requires labeled data for the "source task." The authors suggest that future work could explore how to adapt these post-processors to new domains without requiring new manual annotations.
Summary updated