Harmonious Teacher for Cross-Domain Object Detection

Jinhong Deng, Dongli Xu, Wen Li, Lixin Duan

Abstract

Cross-domain object detection (CDOD) aims to adapt detectors from a labeled source domain to an unlabeled target domain. Traditional self-training methods often suffer from a misalignment between classification confidence and localization accuracy in pseudo-labels, leading to the inclusion of noisy data or the exclusion of valuable hard examples through rigid thresholding. This paper introduces the Harmonious Teacher (HT) framework, which employs a "harmony-based" reweighting strategy and a novel learning objective to align classification and localization signals. By utilizing all pseudo-labels with soft weights, HT achieves new state-of-the-art results on major CDOD benchmarks.

Introduction: The Misalignment Problem

In the context of CDOD, models are typically trained using a "Teacher-Student" paradigm where the Teacher generates pseudo-labels on the target domain. However, a significant issue arises: the classification score is often a poor indicator of localization quality. A bounding box might receive a high classification score despite being poorly aligned with the object, or a well-aligned box might have a lower score.

Comparison of selection criteria: Standard classification scores often favor poorly localized boxes (red), whereas the Harmony Measure identifies better-aligned candidates (blue).

Standard methods rely on hard classification thresholds (e.g., score > 0.9) to filter pseudo-labels. This approach is problematic because it either introduces "disharmonious" noisy boxes into training or discards informative "hard examples" that the model is less certain about but could learn the most from.

Methodology

The Harmonious Teacher framework addresses these issues through two primary innovations: Harmonious Model Learning and Harmonious Sample Reweighting. The implementation is primarily built upon the FCOS (fully convolutional one-stage) detector.

Overview of the Harmonious Teacher framework, illustrating the interaction between Teacher and Student models and the integration of harmonious learning modules.

1. Harmonious Model Learning

To ensure the model's internal signals are consistent, the authors modify the classification branch. Instead of predicting a binary class label, the model is trained to predict the Ground-Truth IoU (GT-IoU) on the source domain. This forces the classification score to serve as a direct estimate of localization quality. On the unlabeled target domain, a self-consistency heuristic is used: the maximum IoU between a predicted box and all other predictions serves as a proxy for localization quality.

2. Harmonious Sample Reweighting

Rather than discarding low-confidence boxes, HT retains all pseudo-labels and assigns them a weight based on a Harmony Measure ($h$): $$h = p^\beta u^{(1-\beta)}$$ where $p$ is the classification score, $u$ is the predicted IoU, and $\beta$ is a balancing hyperparameter. This "soft" reweighting ensures that boxes with both high class confidence and high localization quality have the most influence, while still allowing the model to extract useful information from more difficult, lower-confidence instances.

Experimental Results

The framework was evaluated on four standard benchmarks, consistently outperforming previous state-of-the-art (SOTA) methods:

  • Cityscapes $\to$ Foggy Cityscapes: Achieved 50.4% mAP, a 5% improvement over the previous best (OADA).
  • Cityscapes $\to$ BDD100K: Reached 40.2% mAP, surpassing the top two-stage detector (PT) by 5.3%.
  • Sim10K $\to$ Cityscapes: Achieved 65.5% AP, a significant jump from previous methods which hovered around 55-59%.
  • KITTI $\to$ Cityscapes: Demonstrated robust adaptation across different sensor types and camera setups.

Ablation studies confirmed that the combination of the Harmony Measure for ranking and the Harmonious Weighting loss provides the most substantial performance gains, moving the baseline from 45.2% to 50.4% mAP on the Foggy Cityscapes task.

Conclusion

The Harmonious Teacher framework represents a shift in cross-domain object detection by moving away from binary pseudo-label filtering. By explicitly aligning classification and localization and adopting a continuous reweighting strategy, the model effectively leverages the full spectrum of generated pseudo-labels. While future work may explore its application to Transformer-based architectures like DETR, HT currently sets a new standard for robust domain adaptation in computer vision.

Summary updated

Abstract

Self-training approaches recently achieved promising results in cross-domain object detection, where people iteratively generate pseudo labels for unlabeled target domain samples with a model, and select high-confidence samples to refine the model. In this work, we reveal that the consistency of classification and localization predictions are crucial to measure the quality of pseudo labels, and propose a new Harmonious Teacher approach to improve the self-training for cross-domain object detection. In particular, we first propose to enhance the quality of pseudo labels by regularizing the consistency of the classification and localization scores when training the detection model. The consistency losses are defined for both labeled source samples and the unlabeled target samples. Then, we further remold the traditional sample selection method by a sample reweighing strategy based on the consistency of classification and localization scores to improve the ranking of predictions. This allows us to fully exploit all instance predictions from the target domain without abandoning valuable hard examples. Without bells and whistles, our method shows superior performance in various cross-domain scenarios compared with the state-of-the-art baselines, which validates the effectiveness of our Harmonious Teacher. Our codes will be available at https://github.com/kinredon/Harmonious-Teacher.
BibTeX
@inproceedings{deng2023harmonious,
  author = {Jinhong Deng and Dongli Xu and Wen Li and Lixin Duan},
  title = {{Harmonious Teacher for Cross-Domain Object Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02282},
}
Figure