Object detection models typically output thousands of candidate bounding boxes, many of which cluster around the same physical object. To clean up this redundancy, the computer vision community has long relied on Non-Maximum Suppression (NMS). Standard NMS operates on a simple, greedy "winner-takes-all" principle: it selects the bounding box with the highest confidence score, calculates its Intersection over Union (IoU) with all neighboring boxes, and completely discards any box that exceeds a predefined overlap threshold.
While effective for isolated objects, this hard-thresholding approach breaks down in crowded environments. As noted in the framework proposed for a Dual-Task Integrated Network for Fast Pedestrian Detection in Crowded Scenes, traditional greedy elimination causes a high miss rate when distinct objects naturally overlap. For instance, if two pedestrians are walking closely together, their true bounding boxes will share a high IoU. Standard NMS will inevitably mistake the second pedestrian's bounding box for a redundant detection of the first, erasing a valid object from the final output.
From Deletion to Decay
The core insight of soft non-maximum suppression—originally formalized as Soft-NMS—is that overlap should penalize confidence rather than trigger immediate deletion. Instead of using a step function that drops a box's score to zero the moment it crosses an IoU threshold, soft suppression applies a continuous decay function to the scores of neighboring boxes.
When a high-confidence box is selected, the scores of all overlapping boxes are scaled down proportional to their degree of overlap. The heavily overlapping boxes are kept in the candidate pool, albeit with reduced confidence. If a box still maintains a relatively high score after this penalty, it indicates that the box likely represents a distinct, partially occluded object rather than mere localization noise.
Standard NMS (Hard Threshold):
Score Penalty = 0 if IoU >= Threshold else 1
Soft NMS (Continuous Decay):
Score Penalty = f(IoU) [where f smoothly decreases from 1 to 0 as IoU increases]
Researchers have proposed various mathematical formulations to smooth this decay. For example, Hu demonstrates in Endoscopic Artefact Detection in MMDetection how score decay preserves multiple detections in dense medical imaging regions, preventing the accidental suppression of valid clinical features. To control the severity of the penalty, Dou et al. utilize an exponential decay function in Analysis of vehicle and pedestrian detection effects of improved YOLOv8 model in drone-assisted urban traffic monitoring system, which smoothly scales down confidence based on a Gaussian curve. Alternatively, Zhuang et al. introduce a parameter-free cosine weighting function in SADet: Learning An Efficient and Accurate Pedestrian Detector. This approach creates a smooth transition that naturally suppresses perfectly redundant boxes (where IoU approaches 1) while eliminating the need for manual hyperparameter tuning.
Why Score Decay Preserves Occluded Objects
To understand why this works, consider the underlying signal a detector provides. A bounding box with high overlap and high initial confidence contains two pieces of information: it matches the visual features of the target class, and it is spatially close to another detection. Standard NMS treats spatial proximity as definitive proof of redundancy. Soft suppression treats it as a reason for skepticism.
By decaying the score, the algorithm gives the secondary box a chance to prove its worth. If the secondary box was just a slightly misaligned guess for the same object, its decayed score will fall below the final detection threshold and it will be filtered out. However, if it represents a distinct object—such as closely situated students in a virtual classroom (Shibghatullah et al.), tightly packed items in a freezer warehouse (Yang), or adjacent manufacturing defects (Ruan et al.)—its initial confidence was likely high enough to survive the decay penalty.
Generalizing Beyond 2D Bounding Boxes
The intuition behind score-decaying suppression extends far beyond traditional 2D object detection. Any task that requires selecting discrete intervals or segments from a dense pool of overlapping predictions can benefit from this mechanism.
Temporal Action Localization
In video analysis, actions occur over continuous time intervals rather than spatial boxes. In Multi-Scale Proposal Regression Network for Temporal Action Proposal Generation, Zheng et al. apply a continuous decay function to overlapping temporal proposals. This preservation strategy ensures that high-quality action candidates occurring close together in time are not prematurely discarded during video pruning.
Nested Named Entity Recognition
In natural language processing, linguistic entities are frequently nested inside one another (e.g., "University of California" inside "University of California faculty"). Both Shen et al. in Locate and Label: A Two-stage Identifier for Nested Named Entity Recognition and Wang et al. in An Enhanced Span-based Decomposition Method for Few-Shot Sequence Labeling leverage confidence-decaying span suppression to resolve conflicts between overlapping text spans. This mechanism allows models to rank and extract nested structures that standard filtering would erase.
Hierarchical Classification
When objects belong to nested taxonomic categories, standard suppression can accidentally wipe out valid parent-child relationships. To counter this in marine biology applications, Gupta employs a modified suppression strategy in Hierarchical Object Detection applied to Fish Species that decays scores rather than eliminating overlaps, preserving the co-existence of hierarchical labels for underwater species detection.
Key Foundational Work
A Convnet for Non-maximum Suppression (2016) — contributed 3 concept elements to this direction.
A Convnet for Non-maximum Suppression (2015) — contributed 2 concept elements to this direction.