Spatial Semantic Regularisation for Large Scale Object Detection

Damian Mrowca, Marcus Rohrbach, Judy Hoffman, Ronghang Hu, Kate Saenko, Trevor Darrell

Standard object detection systems have historically excelled at identifying a small number of distinct categories. However, as the field moves toward large-scale detection—distinguishing between thousands of similar classes—traditional methods struggle. When thousands of categories compete for the same pixels, the result is often a cluttered mess of overlapping, contradictory labels. This paper introduces Multi-class Affinity Propagation Clustering (MAPC), a method that leverages semantic hierarchies and spatial overlap to globally reason about detections, significantly reducing false positives and improving localization.

The Limitations of Greedy Suppression

In traditional object detection, the standard post-processing step is Non-Maximum Suppression (NMS). NMS is a simple, greedy heuristic: if two boxes for the same class overlap significantly, the system keeps the one with the highest score and discards the other.

While effective for a few dozen classes, NMS breaks down in large-scale settings for two primary reasons:

  1. Semantic Ambiguity: In a dataset with over 7,000 classes, many labels are nearly identical (e.g., "husky" vs. "malamute" or "couch" vs. "settee"). Because NMS typically operates on a per-class basis, it may leave multiple overlapping boxes of different but related classes on the same object.
  2. Weak Supervision: Large-scale detectors are often trained with limited bounding box data, leading to "noisy" proposals that incorrectly group objects or provide poor localization.
Visualizing the challenge of competing box proposals from different categories and the resulting cleaned detection.

Multi-class Affinity Propagation Clustering (MAPC)

The core contribution of this research is the shift from local NMS to a global regularization framework called MAPC. Instead of treating each class in isolation, MAPC treats all detection proposals in an image as a graph and performs global optimization.

The Similarity Metric

MAPC defines a similarity measure that combines spatial IoU (Intersection over Union) with semantic similarity. The semantic component utilizes the Lin measure, which calculates the distance between two labels within a semantic hierarchy like WordNet. This allows the system to recognize that a "beagle" detection and a "dog" detection are related and should likely be clustered together.

Global Optimization via Message Passing

MAPC uses message-passing to find "exemplars"—the best representative boxes for clusters of similar detections. Unlike standard clustering, MAPC introduces a multi-class constraint: a single physical bounding box location can only be assigned one final label. This prevents the "identity crisis" where a single object is simultaneously labeled as multiple different species.

The MAPC framework uses message passing to identify exemplars across different clusters (A, B, and C) based on spatial and semantic similarity.

Key Insights and Results

By formulating detection as a clustering problem, the system performs global reasoning over the entire image. This allows it to suppress false positives that NMS would miss because they don't overlap "enough" with the top-scoring box but are semantically redundant.

Interestingly, using semantic relationships also improves spatial accuracy. By clustering many overlapping, fine-grained proposals, the "center of gravity" of the cluster often yields a better-localized box than the single highest-scoring raw detection.

Performance Benchmarks

MAPC consistently outperformed NMS across several challenging benchmarks, including Microsoft COCO and ImageNet (1,825 categories):

  • Precision Boost: On the COCO dataset, MAPC achieved a precision of 37.64%, a significant jump over the 23.50% achieved by the standard NMS baseline.
  • F1-Score: The overall F1-score improved from 24.10% to 29.50% on COCO.
  • Error Reduction: The method reduced "wrong label" errors by approximately 7% and "wrong overlap" errors by 23% compared to NMS.

Conclusion

Spatial Semantic Regularisation moves object detection beyond simple greedy heuristics toward a more intelligent, globally-aware system. By acknowledging the relationships between labels, MAPC provides a cleaner and more accurate output for large-scale detection tasks. Future work may extend this logic to the temporal domain, using semantic consistency to stabilize detections across video frames.

Summary updated

BibTeX
@inproceedings{mrowca2015spatial,
  author = {Damian Mrowca and Marcus Rohrbach and Judy Hoffman and Ronghang Hu and Kate Saenko and Trevor Darrell},
  title = {{Spatial Semantic Regularisation for Large Scale Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2015},
  url = {https://openreview.net/forum?id=HJZdugMOZr},
}
Figure