Spatial Semantic Regularisation for Large Scale Object Detection

Damian Mrowca, Marcus Rohrbach, Judy Hoffman, Ronghang Hu, Kate Saenko, Trevor Darrell

Abstract

Large scale object detection with thousands of classes introduces the problem of many contradicting false positive detections, which have to be suppressed. Class-independent non-maximum suppression has traditionally been used for this step, but it does not scale well as the number of classes grows. Traditional non-maximum suppression does not consider label-and instance-level relationships nor does it allow an exploitation of the spatial layout of detection proposals. We propose a new multi-class spatial semantic regularisation method based on affinity propagation clustering, which simultaneously optimises across all categories and all proposed locations in the image, to improve both the localisation and categorisation of selected detection proposals. Constraints are shared across the labels through the semantic WordNet hierarchy. Our approach proves to be especially useful in large scale settings with thousands of classes, where spatial and semantic interactions are very frequent and only weakly supervised detectors can be built due to a lack of bounding box annotations. Detection experiments are conducted on the ImageNet and COCO dataset, and in settings with thousands of detected categories. Our method provides a significant precision improvement by reducing false positives, while simultaneously improving the recall.

Introduction

The field of object detection has seen remarkable progress, yet scaling these systems to handle thousands of object categories remains a significant challenge. A primary bottleneck in large-scale detection is the post-processing stage, where redundant and often contradictory candidate detections must be pruned. Traditionally, this task is handled by Non-Maximum Suppression (NMS), a greedy algorithm that discards overlapping boxes of the same class based on a confidence score.

However, NMS is fundamentally limited in large-scale scenarios. It operates under a class-independent assumption, meaning it fails to recognize semantic relationships between different labels. For instance, if a detector identifies a region as both a "poodle" and a "dog," NMS treats these as independent detections and may retain both, despite them being redundant. As the number of classes increases—often into the thousands—this leads to a "label soup" where images are cluttered with overlapping, semantically related detections that NMS cannot effectively resolve.

Competing box proposals from different categories illustrate the "label soup" problem in large-scale detection.

Multi-class Affinity Propagation Clustering (MAPC)

To address these limitations, we propose Multi-class Affinity Propagation Clustering (MAPC). Instead of the greedy local search employed by NMS, MAPC treats detection selection as a global optimization problem, integrating both spatial and semantic information.

Semantic-Spatial Similarity

The core of MAPC lies in its redefined similarity measure between candidate detections. For any two proposals $i$ and $j$, similarity is calculated by combining their spatial overlap (Intersection over Union, or IoU) with their semantic relatedness. Semantic similarity is derived from the WordNet hierarchy using the Lin measure, which quantifies how closely related two labels are based on their common ancestors in the taxonomy. This allows the model to "know" that a tabby cat and a kitten are more similar than a tabby cat and a microwave.

The Clustering Framework

MAPC utilizes the Affinity Propagation (AP) clustering framework. AP is a message-passing algorithm where data points (detection proposals) exchange "responsibility" and "availability" messages to determine which points should serve as "exemplars"—the final selected detections.

Conceptual diagram of Multi-class Affinity Propagation Clustering (MAPC) showing message passing between clusters based on spatial and semantic similarity.

By optimizing across all categories and locations simultaneously, MAPC introduces global constraints. It ensures that a single physical object is not assigned multiple contradictory labels and employs an "intercluster repellence" term to suppress false positives that do not fit the overall spatial-semantic layout of the image. This global perspective allows the system to recover valid detections that NMS might accidentally suppress; if several low-scoring but semantically related boxes support each other, MAPC can recognize them as a single valid object.

Experimental Evaluation

The authors evaluated MAPC on two major benchmarks: Microsoft COCO and ImageNet. To simulate a large-scale environment on COCO, the standard 80 classes were expanded into 1,133 fine-grained child categories. For ImageNet, the test involved a subset of 1,825 categories with bounding box annotations, drawn from a larger pool of 7,404 categories.

Key Results

The results demonstrate that MAPC significantly outperforms the standard NMS baseline in large-scale settings:

  • Precision Gains: On the fine-tuned COCO dataset, MAPC achieved a precision of 37.64%, a massive jump over the 23.50% achieved by the NMS baseline.
  • Error Reduction: MAPC significantly reduced "wrong label" errors (by ~14% on COCO) and "wrong overlap" errors (by ~20% on COCO), proving that the semantic-spatial clustering successfully groups related detections.
  • F1 Score Improvement: In the non-fine-tuned COCO setup, the F1 score improved from 13.46% to 15.09%.

Discussion and Conclusion

The primary insight of this work is that semantic knowledge can serve as a powerful regulariser for spatial detection problems. By moving beyond greedy selection to a global clustering approach, MAPC prevents the "label soup" that occurs when fine-grained detectors compete for the same pixels. It is particularly effective for weakly supervised models, where bounding box data is scarce and proposals are often poorly localized.

While MAPC offers significant accuracy improvements, it does introduce higher computational complexity due to its message-passing nature compared to the fast, greedy NMS. Future research directions include optimizing the inference time of the Affinity Propagation algorithm and extending the framework to the temporal domain for video object detection, where semantic consistency could be enforced across frames.

Summary updated

Abstract

Large scale object detection with thousands of classes introduces the problem of many contradicting false positive detections, which have to be suppressed. Class-independent non-maximum suppression has traditionally been used for this step, but it does not scale well as the number of classes grows. Traditional non-maximum suppression does not consider label-and instance-level relationships nor does it allow an exploitation of the spatial layout of detection proposals. We propose a new multi-class spatial semantic regularisation method based on affinity propagation clustering, which simultaneously optimises across all categories and all proposed locations in the image, to improve both the localisation and categorisation of selected detection proposals. Constraints are shared across the labels through the semantic WordNet hierarchy. Our approach proves to be especially useful in large scale settings with thousands of classes, where spatial and semantic interactions are very frequent and only weakly supervised detectors can be built due to a lack of bounding box annotations. Detection experiments are conducted on the ImageNet and COCO dataset, and in settings with thousands of detected categories. Our method provides a significant precision improvement by reducing false positives, while simultaneously improving the recall.
BibTeX
@inproceedings{mrowca2015spatial,
  author = {Damian Mrowca and Marcus Rohrbach and Judy Hoffman and Ronghang Hu and Kate Saenko and Trevor Darrell},
  title = {{Spatial Semantic Regularisation for Large Scale Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=HJZdugMOZr},
}
Figure