Detecting an object whose image size is unknown means searching over both position and scale, and every method for doing this cheaply trades accuracy for compute. Depth sensors can predict how big a head should look at each pixel, attentional cascades throw away easy background early, and feature pyramids share computation across scales. Yet papers report each method at one operating point, on their own preprocessing pipeline and their own hardware, with accuracy averaged over all object sizes. Under those conditions nobody can say whether depth-informed scale selection actually beats a well-run image pyramid when both get the same compute, or whether a method's gain sits on near, large targets or on far, small ones.
We propose a locked evaluation harness that measures each scale-search method across a sweep of compute budgets, with recall reported separately for each band of object size. It also fixes the choices that silently change small-object results, such as the interpolation kernel used to build the image pyramid. The scientific claim we test is that budget-matched, size-stratified curves reorder methods relative to the single-number summaries the literature reports. The prediction is that the gain from depth-informed sizing concentrates in the far, small-target band and largely disappears for near targets. If the reordering does not happen, the cheap reporting practice is vindicated, and that is a useful result too.
Core Claim
The clearest evidence that current reporting hides the real trade-off comes from depth-based head detection. Fully Convolutional Network for Head Detection with Depth Images sizes candidate patches from pixel depth and camera focal length under a 200 mm head-width assumption, which removes the image pyramid. It reaches a true positive rate of 0.883 on Watch-n-Patch only with a 3×3 sampling step, running at 0.103 frames per second. A 45×45 step reaches 12.986 fps but "accuracy dropped significantly." The companion version, Head Detection with Depth Images in the Wild, reports the same pattern. Depth removed the scale dimension of the search, but the position dimension still dominates cost.
The two endpoints alone cannot answer two questions. The first is how the curve between them compares to an RGB or depth pyramid given the same number of classifier evaluations. The second is at which target distances the curve falls off.
Our claim is that the right unit of comparison is a per-size-band recall curve as a function of search budget. Let $b$ be the budget, measured both as the number of candidate windows the classifier scores and as latency on one fixed device. Let $s$ index a band of target image height, or of physical distance when depth is available. Let $R_m(b,s)$ be the recall of method $m$ at a fixed false-positive rate. We summarize each method in each band by the normalized area under its budget curve,
$$A_m(s) = \frac{1}{\log b_{\max} - \log b_{\min}} \int_{\log b_{\min}}^{\log b_{\max}} R_m(b,s)\, d\log b ,$$
where the log axis reflects that budgets span two orders of magnitude, as in Ballotta et al. The method contribution is the protocol and the evidence it produces. It is not a new detector.
Relation to Prior Work
The closest work is Segmenting Object Affordances: Reproducibility and Sensitivity to Scale. It retrains affordance segmenters under a fixed resolution, fixed augmentation and a unified metric, and finds that several published models fall below their claimed numbers. It also stress-tests models by simulated zoom and identifies a narrow band of object occupancy outside which accuracy collapses. That paper motivates ours: it shows that standardizing exposes inflated claims and that scale is where models break. Its scope is different from ours in three ways. It evaluates segmentation of single tabletop objects, it never varies compute, and it does not compare scale-search strategies that use geometry. We keep its locked-retraining discipline and add the budget axis, the depth-informed methods and multi-target detection.
The evaluation literature supplies our method. NerfBaselines shows that downscaling kernels, float-versus-uint8 evaluation and background color flip novel-view-synthesis rankings, and it locks these choices by wrapping official code in containers. We reuse that wrapping design and treat pyramid interpolation and bit depth as audited variables. The difference is that our question concerns detection search, not reconstruction. BARS and TFB show that under-tuned baselines and tricks such as dropping the last test batch create apparent progress. From them we take the rule that every method gets the same tuning budget. Open Graph Benchmark shows that realistic splits drop performance 5–8% relative to random splits, which supports our cross-dataset train/test design.
The scale-search literature supplies the methods under test. Depth-based sizing comes from Ballotta et al., and range-segmentation candidate generation from Real-Time Human Detection Based on Range Image Segmentation and Human Detection Using Multiple Classifiers Based on Subtraction Stereo with HOG Features. Early-rejection cascades are represented by An Analysis of the Viola-Jones Face Detection Algorithm. Group Cost-Sensitive Boosting for Multi-Resolution Pedestrian Detection constrains our design. It shows that cascades prematurely reject small pedestrians unless their misses are penalized more heavily, which is exactly the kind of failure an all-sizes average hides. That is why stratification is not optional.
Research Program
The harness wraps each method's official or faithfully reimplemented code in a container. It fixes the following choices: input bit depth (16-bit depth, 8-bit color), pyramid interpolation kernel, scale step, padding policy, matching threshold and metric code. Each method exposes a single budget knob. For sliding-window methods this is the sampling step $K$. For pyramids it is the scale step and stride. For cascades it is the stage rejection threshold. For feature-pyramid detectors it is input resolution and the number of retained proposals.
The harness records candidate count and latency on two fixed devices, a desktop GPU and an embedded board. Latency is reported on both because counting operations alone misses memory and launch overhead.
Ground-truth targets are binned by image height. When depth exists, they are also binned by metric distance and by deviation from the assumed physical size. This tests the rigid 200 mm head assumption directly by sweeping the assumed width. Each method gets an equal hyperparameter search budget on a held-out validation split of the training dataset, never on the test set.
The work proceeds in three phases. Phase one checks fidelity by reproducing published numbers inside the harness. Phase two runs the audit: each published method is evaluated under its original preprocessing and under the locked preprocessing, and the shift in each size band is measured. Phase three produces the comparison itself: budget curves and per-band areas $A_m(s)$ for all methods, with confidence intervals over at least five training seeds. The seed count follows the warning in A Sober Look at the Unsupervised Learning of Disentangled Representations that seed variance can dominate apparent method differences.
Baselines
The naive reference is a dense image pyramid with uniform sliding windows and the same shallow classifier Ballotta et al. use: five convolutional and three fully connected layers on 64×64 patches. It runs on the depth map, and on color where synchronized color frames exist. Every scale-search method is meant to beat this at a fixed budget, so if a method's curve does not lie above it in some band, that method buys nothing there. It runs on the same Pandora-train, Watch-n-Patch-test split as every other method, with the same tuning budget.
The strongest existing depth method is Ballotta et al.'s depth-sized patch extraction. They report a true positive rate of 0.883, IoU 0.635 and 0.103 fps at a 3×3 step on Watch-n-Patch. The prior methods they compare against report 0.709 (Chen et al.) and 0.519 (Nghiem et al.). We must first reproduce 0.883 within seed variation. Beating it means a higher $A_m(s)$ in the far band, or matching 0.883 at a budget at least ten times smaller.
On Caltech Pedestrian the strongest stratified baseline is GCS-LDCF. It reports a 20.20% log-average miss rate on the Reasonable set and 59.53% on the Medium scale. A modern feature-pyramid or anchor-free detector is the most important competitor, because it is the default a practitioner would reach for. There is no reported result on Lacuna for such detectors on Watch-n-Patch or Caltech in this file, so we train one under the same budget sweep rather than quote a figure.
For the dense-prediction extension on Cityscapes, the reference is FarSee-Net. It reports 68.4% mIoU at 84 FPS with 512×768 input on a Titan X, and 21.2 FPS on a Jetson AGX Xavier, according to FarSee-Net. We check whether that figure holds once the input-downscaling kernel is locked.
Two ablations isolate the key ingredient. The first gives the same classifier, at the same candidate count, patches whose size is chosen by depth versus by a pyramid. This separates "depth chooses scale" from "depth chooses where to look." The second holds the method fixed and changes only preprocessing. This isolates protocol drift from algorithmic difference.
Why not just…?
Why not just report small/medium/large AP and frames per second, as detection papers already do? This is the real threat. If coarse size buckets plus one speed number rank methods the same way as our curves, our protocol is unnecessary. The reasons to doubt it are that a single operating point hides the Ballotta-style hundredfold speed range, and that the GCS-Boost result shows small-target failure is concentrated rather than spread across sizes. The decisive experiment computes both summaries from the same runs and counts pairwise rank reversals. If we find none, we will report that the cheap practice suffices.
Why not just retrain everything in one codebase at default settings, as Apicella et al. did? That removes preprocessing drift but not budget confounding. A method evaluated at its authors' preferred step can look better or worse purely because of where it sits on its own curve. The audit phase quantifies how much of the reordering survives locking preprocessing alone, before any budget sweep is added.
Why not count floating-point operations instead of building a latency harness? Operation counts are hardware-independent, but they misrepresent cascades and sliding windows, whose cost is dominated by memory access and early exits. We report both, and we flag any method whose ranking changes between them.
Why not just train with heavy multi-scale augmentation and skip scale search? Apicella et al. found that augmentation "helps, it does not fully resolve this sensitivity." We include an augmentation-trained pyramid-free detector as a baseline so the claim is tested on detection rather than assumed.
Decisive Checks
The program succeeds if three things hold. First, the harness reproduces 0.883 true positive rate on Watch-n-Patch and 20.20% miss rate on Caltech Reasonable within seed intervals, or else documents a specific preprocessing cause for the gap. Second, at least one pairwise ranking between depth-sized, cascade and feature-pyramid methods reverses between the all-sizes summary and the far-band area $A_m(s)$. Third, the depth-versus-pyramid ablation shows the advantage of depth concentrated in the far band and decaying as the assumed physical size deviates from the true one.
A null outcome on the second check is still informative. It would say that existing summaries are adequate and that the field's real bottleneck is spatial sampling density, not scale.
References
- Fully Convolutional Network for Head Detection with Depth Images
- Head Detection with Depth Images in the Wild
- Segmenting Object Affordances: Reproducibility and Sensitivity to Scale
- NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods
- BARS: Towards Open Benchmarking for Recommender Systems
- TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods
- Open Graph Benchmark: Datasets for Machine Learning on Graphs
- Real-Time Human Detection Based on Range Image Segmentation
- Human Detection Using Multiple Classifiers Based on Subtraction Stereo with HOG Features
- An Analysis of the Viola-Jones Face Detection Algorithm
- Group Cost-Sensitive Boosting for Multi-Resolution Pedestrian Detection
- A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation
- FarSee-Net: Real-Time Semantic Segmentation by Efficient Multi-scale Context Aggregation and Feature Space Super-resolution