Choosing Supervised Post-Processing for Frozen Person Detectors: Information, Object Changes, and Complete Cost

Tianyi Zhou, Zijun JiaarXiv 2026 · arXiv

TL;DR

Supervised post-processing can significantly boost the performance of "frozen" object detectors (where weights cannot be changed) by re-evaluating discarded hypotheses. This paper demonstrates that while richer feature sets like "Full100" improve detection, the real-world cost depends on implementation optimization rather than feature dimensionality, and improved performance often involves a complex "churn" of gaining new objects while losing some previously correct ones.

What Problem Does This Paper Address?

Standard object detectors (like Faster R-CNN) generate thousands of candidate boxes (hypotheses), but most are thrown away during a step called Non-Maximum Suppression (NMS) to prevent overlapping duplicates. Some of these discarded boxes actually contain real objects that the detector "saw" but failed to prioritize.

In many practical scenarios, you can't "fine-tune" the detector—perhaps you're accessing it via a fixed API or using a specialized hardware accelerator. This is a frozen detector. The authors ask: If we can't change the detector, how much can we improve its output just by smarter post-processing? Specifically, they investigate which pieces of information (scores, box geometry, or context) are worth the computational cost of processing.

Methods

The authors propose a framework for Supervised Post-Processing that treats the detector's internal outputs as features for a secondary, lightweight classifier (a Histogram Gradient-Boosted tree, or HGB).

Figure 1: Study and deployment interface. (A) Source labels train readouts and select scalar maps and thresholds offline. (B) A request executes one chosen policy with one fixed-detector forward and its required outputs. Full100 and meta-statistics readouts are alternatives. (C) Held-out evaluation compares complete outputs and processing costs.

Key components include:

  • The Full100 Readout: A 100-dimensional feature vector extracted for every candidate box. It includes 91 class probabilities, 6 geometric terms (coordinates, area, aspect ratio), and 3 "native-set summaries" that describe how a candidate relates to the detector's default, high-confidence outputs.
  • Comparison Models: They compare Full100 against simpler baselines (like using only person/background scores) and established methods like MetaDetect, which uses neighborhood statistics (how many other boxes are nearby).
  • Execution Optimization: To ensure a fair "complete cost" comparison, they optimized the neighborhood statistics calculations (C99 implementation) to see if complex features are inherently slower.

Datasets

The study uses several challenging benchmarks to test the limits of person detection:

  • CrowdHuman: The primary dataset, known for extreme overlapping and dense crowds.
  • MOT17Det: Used to test "source-frozen transfer"—seeing if a post-processor trained on CrowdHuman works on video frames without further tuning.
  • MIAP (More Inclusive People Annotations): A subset of Open Images used to test how the models handle different demographic distributions.

Key Insights and Ideas

  • Context is King: The most significant performance gains didn't come from having more class probabilities, but from "native-set context"—knowing how a candidate box overlaps with the boxes the detector already decided were good.
  • The "Churn" of Discoveries: A higher True Positive (TP) count doesn't mean the new model is a strict upgrade. As shown in the figure below, Full100 gains many new objects but also "loses" some that the raw detector originally caught. This "replacement" behavior is critical for engineers to understand before deploying a new policy.
Figure 3: Objects gained and lost by Full100 relative to strong references. Each row uses actual matched annotation identities; CH-319 and MOT17Det are distinct cohorts.
  • Dimension $\neq$ Cost: A common assumption is that more features = slower code. The authors found that after optimization, the 141-dimensional MetaStats model was nearly as fast as the 100-dimensional Full100 model (~95ms). The bottleneck is often data transfer and setup, not the number of features.
  • Feature Non-Monotonicity: Simply adding more features doesn't always help. For instance, using all 91 class slots sometimes performed worse than just using the person/background scores unless geometry and context were also included.

Key Results

  • Detection Boost: On CrowdHuman, the Full100 model matched 6,004 objects, a significant jump over the optimized raw baseline (5,848 objects).
  • Efficiency: The "complete warm cost" (the time from reading an image to final output) for Full100 was approximately 95 ms on an NVIDIA A800 GPU. This represents only a ~6-7% overhead compared to raw processing, making it a viable "drop-in" improvement.
  • Transferability: While the model trained on CrowdHuman transferred well to MOT17Det (showing a gain in True Positives), it showed no improvement on MIAP, suggesting that post-processing is sensitive to the specific "look" and box geometry of the training data.

Limitations and Future Work

  • Task Specificity: The study focuses exclusively on person detection. Results might differ for multi-class datasets with very different object scales (e.g., COCO).
  • Hardware Dependency: Latency measurements are specific to the tested hardware (NVIDIA A800). Performance on edge devices might favor simpler feature sets.
  • Label-Free Adaptation: The current method requires labeled data for the "source task." The authors suggest that future work could explore how to adapt these post-processors to new domains without requiring new manual annotations.

Summary updated

Abstract

A frozen detector computes hypotheses that can support better final outputs through supervised post-processing. Choosing a readout requires knowing which available information improves detection and what its processing costs. We compare candidate scores, class distributions, geometry, native-set summaries and adapted neighborhood meta classification under common candidate, supervision and operating-point conditions. On 320 held-out CrowdHuman images, adding native-set context to a 97-dimensional readout yields 275 more matches and 96 more unmatched predictions. On a separate 319-image cohort, Full100 matches 6004 objects,versus 5848 for source-selected raw WBF and 6002 for a MetaDetect-style adaptation. Its gain over raw contains 300 added and 144 lost discoveries. After execution-equivalent optimization, Full100 and the meta-statistics readout both take approximately 95 ms per complete warm request on a fixed 64-image workload. Source-frozen transfer gives positive point results on 223 MOT17Det frames from seven sequences, while the main MIAP result shows no improvement. The study supports using sufficiently informative output readouts while retaining optimized meta classification as a competitive option. It also shows why feature dimension and net detection gain alone are unreliable guides to adoption.
BibTeX
@misc{zhou2026choosing,
  author = {Tianyi Zhou and Zijun Jia},
  title = {{Choosing Supervised Post-Processing for Frozen Person Detectors: Information, Object Changes, and Complete Cost}},
  year = {2026},
  eprint = {2611.24003},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24003},
}

Loading related content...

Figure