TL;DR
This paper introduces "Mutual Supervision" (MuSu), a novel training strategy for dense object detectors that breaks the convention of using identical training samples for classification and regression heads. By allowing these heads to mutually supervise each other, MuSu significantly improves detection accuracy and consistency, particularly in challenging scenarios.
What Problem Does This Paper Address?
Dense object detectors, like FCOS, rely on two main components: a classification head to identify objects and a regression head to pinpoint their precise locations. Traditionally, both heads are trained using the same set of samples, leading to a potential inconsistency where the highest classification score and the most accurate bounding box don't always align for the same object. This misalignment can cause good detections to be suppressed during post-processing (like Non-Maximum Suppression or NMS), ultimately hurting performance. The paper aims to bridge this gap by ensuring better alignment and consistency between these two crucial heads.
Methods
The core innovation is the "Mutual Supervision" (MuSu) algorithm. Instead of sharing training samples, MuSu defines them adaptively and reciprocally for each head:
- Adaptive Candidate Bags: For each ground-truth object, MuSu first constructs a "candidate bag" of potential anchors. This bag is formed by considering a joint likelihood of an anchor's classification score and its IoU with the ground-truth box. This step filters out obviously incorrect anchors early on.
- Mutual Assignment:
- The classification head is trained using samples that are selected based on the localization scores (IoU) from the regression head.
- The regression head is trained using samples selected based on the classification scores from the classification head.
- Soft Targets and Weighting: MuSu treats training samples as "soft targets." It assigns weights to the losses of anchors within the candidate bags. These weights are determined by the ranking of anchors based on the counterpart head's prediction quality. Anchors that perform better according to the other head receive higher weights. This is achieved through a negative exponential function of the rank.
- Regularized Criteria: To compute these rankings, MuSu uses a regularized criterion that combines the classification score ($p_i$) and IoU ($q_i$) of an anchor, controlled by a hyper-parameter $\alpha$. This allows for a more robust and adaptive assignment than using either score alone.
Datasets
- MS COCO: The primary dataset used for training and evaluation. Experiments were conducted on the
trainval35ksubset for training and theminivalsubset for ablation studies. Final results were reported on thetest-devsubset.
Key Insights and Ideas
- Inconsistency is a Bottleneck: The paper convincingly demonstrates that the misalignment between classification and regression heads is a significant performance bottleneck, even in converged detectors.
- Mutual Supervision Works: The core idea of having heads supervise each other based on the other's output is highly effective. It forces a more holistic understanding of object detection.
- Soft Targets are Key: Using soft targets (weighted losses) rather than hard assignments allows for more nuanced supervision and better adaptation to challenging cases.
- Adaptive Sample Selection is Powerful: MuSu's ability to dynamically select training samples based on the current state of both heads makes it robust and adaptable.
- Tiling More Anchors Benefits MuSu: Unlike some prior methods, MuSu effectively leverages the strategy of tiling more anchors per location, leading to further performance gains. This suggests MuSu can better exploit the richer information provided by denser anchor settings.
Key Results
- Significant AP Improvement: MuSu boosted the FCOS detector with a ResNet-50 backbone to 40.6 AP on the MS COCO validation set, a substantial improvement over the baseline FCOS (36.5 AP) and even improved FCOS++ (38.6 AP).
- State-of-the-Art Performance: When using deeper backbones (ResNet-101 and ResNet-101-DCN), MuSu achieved competitive results, surpassing previous methods in AP75 and being on par with other state-of-the-art approaches like PAA.
- Tiling Anchors Boosts Performance: With more anchors per location (e.g., #A=3), MuSu achieved 40.9 AP, outperforming baselines and demonstrating its ability to leverage denser anchor settings.
- No Inference Speed Sacrifice: MuSu is a training-time strategy and does not add any new modules, thus maintaining the inference speed of the original detector (for #A=1).
Limitations and Future Work
- Computational Complexity: While MuSu doesn't add parameters, using multiple anchors per location increases computational cost during NMS, leading to a slight drop in FPS.
- Hyperparameter Tuning: The paper explores various settings for the regularization parameter $\alpha$ and temperature coefficients ($T_{cls}, T_{reg}$), suggesting that careful tuning can further optimize performance.
- Generalizability: The paper suggests that MuSu is a general framework compatible with various training sample selection methods and could be extended to other detector architectures. Future work could explore these extensions.
- Soft vs. Hard Targets: While soft targets performed better, the paper notes that hard targets yielded a respectable 40.0 AP, indicating potential for simpler implementations.
Summary updated