Disentangle Your Dense Object Detector

Zehui Chen, Chenhongyi Yang, Qiaofei Li, Feng Zhao, Zheng-Jun Zha, Feng Wu

TL;DR

The Disentangled Dense Object Detector (DDOD) identifies and breaks three hidden "conjunctions" in standard single-stage detector training: label assignment, spatial features, and scale-wise supervision. By decoupling these tasks, DDOD achieves significant accuracy gains (up to 2.4 mAP) across various architectures with almost zero impact on inference speed.

What Problem Does This Paper Address?

Single-stage (dense) object detectors like RetinaNet or FCOS are popular because they are fast and simple. However, their training pipelines often rely on "conjunctions"—assumptions that classification and bounding-box regression should be treated identically.

The authors argue that this "one-size-fits-all" approach is a bottleneck. Specifically, they identify three areas where entanglement hurts performance:

  1. Label Assignment: Usually, if a sample is "positive" for classification, it’s automatically used for regression. But the best samples for identifying what an object is aren't always the best for determining where its boundaries are.
  2. Spatial Features: Classification and regression branches typically share the same input features and receptive fields. However, classification needs to see the "heart" of an object (semantic features), while regression needs to see the "edges" (geometric features).
  3. Pyramid Supervision: Modern detectors use Feature Pyramid Networks (FPN) to detect objects at different scales. Because lower layers have many more pixels than higher layers, the training process becomes biased toward small objects, leaving large objects under-supervised.

Methods

The authors propose DDOD, a training framework that introduces three specific disentanglement mechanisms:

  • Label Assignment Disentanglement: Instead of a shared set of positive samples, DDOD uses a cost-based assigner that independently selects the best candidates for classification and regression. It uses a hyperparameter ($\alpha$) to tune the balance between classification scores and localization accuracy for each branch.
  • Spatial Feature Disentanglement: The authors replace standard convolutions in the detection heads with Deformable Convolutions (DCN). This allows the model to adaptively "shift" its gaze. The classification head learns to focus on discriminative parts (like a face), while the regression head focuses on boundaries (like the top of a head).
  • Pyramid Supervision Disentanglement: To fix the scale imbalance, they introduce a FPN Hierarchical Loss. This mechanism re-weights the loss for each layer based on the number of positive samples it contains, ensuring that the sparse signals from large objects in high-level layers aren't drowned out by the noisy signals from small objects in low-level layers.

Datasets

The researchers evaluated DDOD on two major benchmarks:

  • MS COCO: The gold standard for general object detection.
  • WIDER FACE: A challenging dataset specifically for face detection, characterized by extreme variations in scale and occlusion.

Key Insights and Ideas

  • The "Aha" Moment on Labels: Pilot experiments showed that the optimal IoU threshold for classification is higher (0.5) than for regression (0.4). This proves that the regression branch actually benefits from seeing "looser" samples that the classification branch would consider background.
  • Visualizing the Split: Using GradCAM, the authors visualized that the classification branch naturally wants to look at the center of objects, while the regression branch wants to look at the edges. Forcing them to share the exact same pixels via standard convolutions creates a "tug-of-war" that limits both.
  • Free Lunch Performance: Because these changes primarily affect the training phase (with the exception of a single deformable convolution layer), the performance boost comes with "negligible extra overhead" during actual use (inference).

Key Results

DDOD consistently improved every baseline it was tested against:

  • RetinaNet: +2.0 mAP improvement.
  • FCOS: +2.4 mAP improvement.
  • ATSS: +2.2 mAP improvement.
  • State-of-the-Art: Their best model reached 55.0 mAP on the COCO test-dev set and 93.5 AP on the WIDER FACE "Hard" subset, setting new benchmarks at the time of publication.
  • Efficiency: The inference time increase was minimal (e.g., moving from 51ms to 54ms), making it highly practical for real-world deployment.

Limitations and Future Work

While DDOD is highly effective for "dense" (single-stage) detectors, the authors note that the problem of task entanglement likely exists in "sparse" (two-stage) detectors like Faster R-CNN as well. The current implementation focuses on the parallel head architecture common in single-stage models.

Future work suggested by the authors includes exploring how to embed these disentanglement principles into R-CNN-like architectures to see if similar "free" gains can be achieved in multi-stage detection pipelines.

Summary updated

Abstract

Deep learning-based dense object detectors have achieved great success in the past few years and have been applied to numerous multimedia applications such as video understanding. However, the current training pipeline for dense detectors is compromised to lots of conjunctions that may not hold. In this paper, we investigate three such important conjunctions: 1) only samples assigned as positive in classification head are used to train the regression head; 2) classification and regression share the same input feature and computational fields defined by the parallel head architecture; and 3) samples distributed in different feature pyramid layers are treated equally when computing the loss. We first carry out a series of pilot experiments to show disentangling such conjunctions can lead to persistent performance improvement. Then, based on these findings, we propose Disentangled Dense Object Detector (DDOD), in which simple and effective disentanglement mechanisms are designed and integrated into the current state-of-the-art dense object detectors. Extensive experiments on MS COCO benchmark show that our approach can lead to 2.0 mAP, 2.4 mAP and 2.2 mAP absolute improvements on RetinaNet, FCOS, and ATSS baselines with negligible extra overhead. Notably, our best model reaches 55.0 mAP on the COCO test-dev set and 93.5 AP on the hard subset of WIDER FACE, achieving new state-of-the-art performance on these two competitive benchmarks. Code is available at https://github.com/zehuichen123/DDOD.
BibTeX
@inproceedings{chen2021disentangle,
  author = {Zehui Chen and Chenhongyi Yang and Qiaofei Li and Feng Zhao and Zheng-Jun Zha and Feng Wu},
  title = {{Disentangle Your Dense Object Detector}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475351},
}

Loading related content...

Figure