Mutual Supervision for Dense Object Detection

Ziteng Gao, Limin Wang, Gangshan Wu

TL;DR

This paper introduces "Mutual Supervision" (MuSu), a novel training strategy for dense object detectors that breaks the convention of using identical training samples for classification and regression heads. By allowing these heads to mutually supervise each other, MuSu significantly improves detection accuracy and consistency, particularly in challenging scenarios.

What Problem Does This Paper Address?

Dense object detectors, like FCOS, rely on two main components: a classification head to identify objects and a regression head to pinpoint their precise locations. Traditionally, both heads are trained using the same set of samples, leading to a potential inconsistency where the highest classification score and the most accurate bounding box don't always align for the same object. This misalignment can cause good detections to be suppressed during post-processing (like Non-Maximum Suppression or NMS), ultimately hurting performance. The paper aims to bridge this gap by ensuring better alignment and consistency between these two crucial heads.

Methods

The core innovation is the "Mutual Supervision" (MuSu) algorithm. Instead of sharing training samples, MuSu defines them adaptively and reciprocally for each head:

  • Adaptive Candidate Bags: For each ground-truth object, MuSu first constructs a "candidate bag" of potential anchors. This bag is formed by considering a joint likelihood of an anchor's classification score and its IoU with the ground-truth box. This step filters out obviously incorrect anchors early on.
  • Mutual Assignment:
    • The classification head is trained using samples that are selected based on the localization scores (IoU) from the regression head.
  • The regression head is trained using samples selected based on the classification scores from the classification head.
  • Soft Targets and Weighting: MuSu treats training samples as "soft targets." It assigns weights to the losses of anchors within the candidate bags. These weights are determined by the ranking of anchors based on the counterpart head's prediction quality. Anchors that perform better according to the other head receive higher weights. This is achieved through a negative exponential function of the rank.
  • Regularized Criteria: To compute these rankings, MuSu uses a regularized criterion that combines the classification score ($p_i$) and IoU ($q_i$) of an anchor, controlled by a hyper-parameter $\alpha$. This allows for a more robust and adaptive assignment than using either score alone.

Datasets

  • MS COCO: The primary dataset used for training and evaluation. Experiments were conducted on the trainval35k subset for training and the minival subset for ablation studies. Final results were reported on the test-dev subset.

Key Insights and Ideas

  • Inconsistency is a Bottleneck: The paper convincingly demonstrates that the misalignment between classification and regression heads is a significant performance bottleneck, even in converged detectors.
  • Mutual Supervision Works: The core idea of having heads supervise each other based on the other's output is highly effective. It forces a more holistic understanding of object detection.
  • Soft Targets are Key: Using soft targets (weighted losses) rather than hard assignments allows for more nuanced supervision and better adaptation to challenging cases.
  • Adaptive Sample Selection is Powerful: MuSu's ability to dynamically select training samples based on the current state of both heads makes it robust and adaptable.
  • Tiling More Anchors Benefits MuSu: Unlike some prior methods, MuSu effectively leverages the strategy of tiling more anchors per location, leading to further performance gains. This suggests MuSu can better exploit the richer information provided by denser anchor settings.

Key Results

  • Significant AP Improvement: MuSu boosted the FCOS detector with a ResNet-50 backbone to 40.6 AP on the MS COCO validation set, a substantial improvement over the baseline FCOS (36.5 AP) and even improved FCOS++ (38.6 AP).
  • State-of-the-Art Performance: When using deeper backbones (ResNet-101 and ResNet-101-DCN), MuSu achieved competitive results, surpassing previous methods in AP75 and being on par with other state-of-the-art approaches like PAA.
  • Tiling Anchors Boosts Performance: With more anchors per location (e.g., #A=3), MuSu achieved 40.9 AP, outperforming baselines and demonstrating its ability to leverage denser anchor settings.
  • No Inference Speed Sacrifice: MuSu is a training-time strategy and does not add any new modules, thus maintaining the inference speed of the original detector (for #A=1).

Limitations and Future Work

  • Computational Complexity: While MuSu doesn't add parameters, using multiple anchors per location increases computational cost during NMS, leading to a slight drop in FPS.
  • Hyperparameter Tuning: The paper explores various settings for the regularization parameter $\alpha$ and temperature coefficients ($T_{cls}, T_{reg}$), suggesting that careful tuning can further optimize performance.
  • Generalizability: The paper suggests that MuSu is a general framework compatible with various training sample selection methods and could be extended to other detector architectures. Future work could explore these extensions.
  • Soft vs. Hard Targets: While soft targets performed better, the paper notes that hard targets yielded a respectable 40.0 AP, indicating potential for simpler implementations.

Summary updated

Abstract

The classification and regression head are both indispensable components to build up a dense object detector, which are usually supervised by the same training samples and thus expected to have consistency with each other for detecting objects accurately in the detection pipeline. In this paper, we break the convention of the same training samples for these two heads in dense detectors and explore a novel supervisory paradigm, termed as Mutual Supervision (MuSu), to respectively and mutually assign training samples for the classification and regression head to ensure this consistency. MuSu defines training samples for the regression head mainly based on classification predicting scores and in turn, defines samples for the classification head based on localization scores from the regression head. Experimental results show that the convergence of detectors trained by this mutual supervision is guaranteed and the effectiveness of the proposed method is verified on the challenging MS COCO benchmark. We also find that tiling more anchors at the same location benefits detectors and leads to further improvements under this training scheme. We hope this work can inspire further researches on the interaction of the classification and regression task in detection and the supervision paradigm for detectors, especially separately for these two heads.
BibTeX
@inproceedings{gao2021mutual,
  author = {Ziteng Gao and Limin Wang and Gangshan Wu},
  title = {{Mutual Supervision for Dense Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00362},
}
Figure