Object detectors propose many overlapping candidate boxes and then run a cleanup step, non-maximum suppression (NMS), that keeps one box per object and deletes the rest. In crowded scenes, such as packed sidewalks, traffic jams, or retail shelves, two real people can overlap as much as two duplicate guesses about one person. Distance-aware NMS handles this by also checking how far apart the box centers are, not just how much the boxes overlap. That check only works if the detector's boxes are tight and well centered. Structured pruning removes whole channels, attention heads, or blocks to make a detector run faster on real hardware, and it is tuned on the network's training loss and on average-case benchmarks. Nobody checks whether pruning quietly loosens the box geometry that crowd-aware cleanup depends on.
We propose to test the hypothesis that latency-constrained pruning damages crowded-scene detection more than its average-case accuracy suggests. The proposed mechanism is that pruning erodes the geometric margin separating duplicate box pairs from neighbor box pairs, and we expect the size of this erosion to depend on the backbone family (convolutional versus attention-based). If the effect is real, the fix is to make pruning "suppression-aware": score each removable structure by how much it contributes to that margin, and charge the cleanup step's own runtime against the latency budget on edge devices where that runtime is not negligible.
Core Claim
Hardware-aware pruning methods measure and constrain what the network costs, but they judge accuracy only after a standard IoU cleanup step on sparse benchmarks. HALP prunes CNN filters against measured GPU latency tables, and its SSD experiment touches only the backbone. MDP extends latency-table pruning to CNN channels, Transformer heads, and whole blocks through one mixed-integer program, and evaluates detection on Pascal VOC and NuScenes. SAViT prunes Vision Transformers jointly across components under FLOP budgets and reports a 0.3 mAP drop for a Swin-T Faster R-CNN on COCO at 70% FLOP reduction. None of the three evaluates on dense crowds or uses distance-aware cleanup.
On the cleanup side, Adaptive NMS explicitly warns that raising the suppression threshold keeps duplicates when boxes are not compact. The only paper in the file that combines pruning with distance-aware NMS, Xiao et al.'s ship detector, prunes YOLOv3 channels and treats the cleanup step as a fixed plug-in on a sparse SAR dataset. That paper therefore motivates the question without answering it.
Our claim is that the right object to preserve under pruning is a pairwise quantity, the separability of duplicate pairs from neighbor pairs under the chosen suppression metric, and that this quantity can be added to an existing pruning program as a saliency term. Concretely, let $M_{ij}$ be the DIoU suppression score between predicted boxes $i$ and $j$, defined as IoU minus squared center distance over squared enclosing-box diagonal, following Distance-IoU. Let $N_t$ be the suppression threshold. Let $\mathcal{D}$ be the set of high-scoring prediction pairs matched to the same ground-truth object, and $\mathcal{N}$ the set of pairs matched to different, overlapping ground-truth objects. We define
$$ L_{sep} = \sum_{(i,j)\in\mathcal{D}} \max(0,\, N_t + \delta - M_{ij}) + \sum_{(i,j)\in\mathcal{N}} \max(0,\, M_{ij} - N_t + \delta), $$
where $\delta$ is a small margin. $L_{sep}$ is zero when every duplicate pair would be suppressed and every neighbor pair kept with room to spare. Because $M_{ij}$ is a smooth function of box coordinates, the first-order Taylor change in $L_{sep}$ from masking a structure $s$ is computable from one backward pass. This is the same kind of estimate HALP uses for task-loss saliency. We add this term to the saliency of each structure in MDP's program.
Research Program
The work proceeds in three stages, and each stage gates the next.
The first stage is diagnostic and is the decisive part. We prune three detector families to matched measured-latency targets with MDP as published. The families are a ResNet-50 FPN Faster R-CNN, a Swin-T Faster R-CNN (the SAViT setting), and DETA, the Deformable-DETR variant from NMS Strikes Back that keeps one-to-many assignment and NMS. On CrowdHuman we then measure the dense and pruned models on three geometric quantities: center error, box tightness, and the distributions of $M_{ij}$ over $\mathcal{D}$ and $\mathcal{N}$, summarized as a separability AUC. We also measure the accuracy gap between IoU-NMS and DIoU-NMS before and after pruning. The hypothesis predicts two things. First, crowd miss rate should degrade proportionally more than COCO AP at equal latency. Second, the benefit of DIoU-NMS over IoU-NMS should shrink after pruning. If neither shows up, the project stops at a useful negative result: pruning is safe for crowd cleanup.
The second stage adds the separability saliency to MDP and uses $L_{sep}$ during recovery fine-tuning. The study asks which structures carry crowd geometry. Candidates are late convolutional stages versus FFN width versus attention heads, and the answer may differ by family. This is the cross-architecture question that MDP's unified encoding makes answerable without custom heuristics per block type.
The third stage treats cleanup latency as a constraint, not as the headline. Latency NMS Attacks shows that NMS time is negligible on server GPUs and is bounded by the usual 300-detection cap, but grows on edge CPUs. That paper benchmarks only IoU NMS. We therefore profile IoU-NMS, DIoU-NMS, and Adaptive-NMS as a function of candidate count on Jetson AGX Orin and an ARM CPU. We fold the fitted cost into the latency budget using each pruned model's measured candidate count on a calibration set. If this term turns out to be small relative to network latency, we will report that and drop it.
Baselines
The naive reference point is the dense detector with each cleanup rule (IoU-NMS, DIoU-NMS, Adaptive-NMS) at its tuned threshold. The comparable published crowd number is Adaptive-NMS with FPN on CrowdHuman, reported in Adaptive NMS, which lowers the miss rate (MR⁻², a log-average miss rate) from 52.35% to 49.73%. On CityPersons the same paper reports 10.8% versus 15.4% for baseline Faster R-CNN. These numbers anchor whether our dense baselines are trained competently.
The strongest existing pruning method is MDP, because it already handles both CNN and Transformer structures under measured latency. We run it with the authors' formulation, the same latency tables measured on our devices, the same latency targets, and the same fine-tuning epochs and hyperparameter search budget as our variant. We also give it the best per-model re-tuned NMS threshold, so that any gain we report is not just threshold drift. We found no reported result on Lacuna for MDP, SAViT, or HALP on CrowdHuman or CityPersons. For context we cite SAViT's 0.3 mAP COCO drop at 70% FLOP reduction for Swin-T Faster R-CNN and HALP's 1.94× throughput at 0.56 mAP drop for SSD on VOC.
We also include FLOP-budgeted SAViT-style pruning at matched measured latency, to separate the latency-table effect from the separability effect.
The key ablation runs MDP plus recovery fine-tuning with standard output distillation from the dense model. We test it with and without $L_{sep}$ in the saliency and in the fine-tuning loss, so that we can tell whether the pairwise margin matters beyond generic box imitation.
Beating the baselines means a lower CrowdHuman MR⁻² at equal measured end-to-end latency on at least two of the three families, a COCO AP within noise of MDP, and a separability AUC that tracks the miss-rate gain.
Why not just…?
Why not just drop NMS with an end-to-end detector like YOLOv10? YOLOv10 removes the cleanup step through dual label assignment and is fast on COCO. It also acknowledges a gap to NMS-based models at its smallest sizes, and it does not report crowd results; we found no reported result on Lacuna for YOLOv10 on CrowdHuman. DETA's results argue that one-to-many training plus NMS remains competitive. We include YOLOv10-S and YOLOv10-M at matched latency on CrowdHuman. If they beat every pruned NMS model, that is a real threat, and we would report it as the answer.
Why not just re-tune the NMS threshold after pruning, or switch to Soft-NMS? Re-tuning is cheap, and it is built into our MDP baseline. It cannot, however, restore separation once the duplicate and neighbor score distributions overlap, and the stage-one AUC measurement shows directly whether they do. Soft-NMS is reported in Adaptive NMS to behave like greedy NMS at high density, and we include it as a cleanup option anyway.
Why not just distill the dense model's boxes during fine-tuning? This is the most plausible competitor. Box-level distillation may restore tightness without any pairwise term, so it is our central ablation. If distillation alone matches the full method, the honest conclusion is that the contribution is the diagnostic finding plus a recommendation to distill, and the saliency term is unnecessary.
Why not just add a constant NMS cost to the latency table? On GPUs with the 300-detection cap this is probably sufficient, and we expect it to be. The candidate-dependent model matters only if pruned models emit measurably more candidates on edge CPUs. Stage three measures this directly, and the term is dropped if it does not.
Proposed Experiments
The decisive check is the stage-one plot, drawn for each family. It shows crowd MR⁻² degradation against COCO AP degradation at matched Orin latency, with separability AUC as a third axis. We test the hypothesis that MR⁻² degrades disproportionately and that AUC predicts it.
Stage two then reports CrowdHuman (full-body and visible) and CityPersons MR⁻², together with COCO AP, at three latency targets. Latency is measured end to end, including cleanup, on an A100, a Jetson AGX Orin, and an ARM CPU. We include per-structure saliency maps showing which components carry crowd geometry in CNN versus Swin versus DETA models.
A secondary check pre-trains on CrowdHuman and transfers to CityPersons. This follows the CrowdHuman paper's reported improvement from 14.81% to 10.67% mMR and tests whether separability-aware pruning preserves that transfer benefit.
References
- HALP: Hardware-Aware Latency Pruning
- MDP: Multidimensional Vision Model Pruning with Latency Constraint
- SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization
- Adaptive NMS: Refining Pedestrian Detection in a Crowd.
- A Tiny Model for Fast and Precise Ship Detection via Feature Channel Pruning
- Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression
- NMS Strikes Back
- Latency NMS Attacks: Is It Real Life or Is It Just Fantasy?
- YOLOv10: Real-Time End-to-End Object Detection