Improving Object Detection by Label Assignment Distillation

Chuong H. Nguyen, Thuy C. Nguyen, Tuan N. Tang, Nam L.h. PhanWACV 2022

TL;DR

This paper introduces Label Assignment Distillation (LAD), a novel distillation technique for object detection that improves label assignment by leveraging a teacher network. LAD is shown to be effective even with smaller teachers and can be combined with existing methods, leading to significant performance gains on the COCO benchmark.

What Problem Does This Paper Address?

Object detection, a fundamental computer vision task, relies on assigning "labels" to image regions to indicate whether they contain an object (foreground) or not (background). Unlike image classification, where this is straightforward, object detection faces ambiguity due to overlapping bounding boxes. Existing methods for assigning these labels are often static and heuristically designed. While distillation techniques are common for transferring knowledge from a larger "teacher" model to a smaller "student" model, they typically focus on mimicking the teacher's direct outputs. This paper argues that the label assignment process itself is a rich source of knowledge that can be distilled, and that current methods don't fully exploit this.

Methods

The core contribution is Label Assignment Distillation (LAD). Instead of directly using a teacher's predictions as targets (soft-label distillation), LAD uses the teacher's predictions to inform the cost function used for label assignment. This cost function then guides the student's learning process. The authors illustrate this with the Probabilistic Anchor Assignment (PAA) method:

  • Soft-Label Distillation (Fig. 1a): The teacher's predicted class probabilities and bounding boxes are used directly as targets for the student.
  • Label Assignment Distillation (LAD) (Fig. 1b): The teacher's predictions are used to calculate an assignment cost. This cost is then used to determine the "ground truth" labels for the student, effectively distilling the teacher's label assignment strategy.

The paper also proposes Co-learning Label Assignment Distillation (CoLAD). This method allows two networks to be trained simultaneously from scratch, dynamically switching roles as teacher and student based on a performance indicator (e.g., standard deviation over mean of assignment costs). This avoids the need for a pre-trained teacher and can help escape local minima.

Datasets

The experiments were conducted on the MS COCO benchmark. This dataset contains approximately 118k images for training, 5k for validation, and 20k for testing. The authors report ablation studies on the validation set and compare their final results on the test-dev set.

Key Insights and Ideas

  • Label assignment is a rich source of knowledge: The paper highlights that the process of deciding which regions are foreground/background is crucial and can be effectively distilled, not just the final predictions.
  • LAD is more effective than soft-label distillation for label assignment: Experiments show LAD outperforms soft-label distillation, especially when using a smaller teacher.
  • LAD enables bidirectional enhancement: A smaller teacher can significantly improve a larger student, and remarkably, the student can even surpass its teacher. This suggests LAD offers a unique form of knowledge transfer.
  • CoLAD removes the need for a pre-trained teacher: By allowing networks to learn from each other dynamically, CoLAD offers a more flexible and potentially more robust training paradigm.
  • Dynamic switching is key: The CoLAD framework's ability to switch teacher-student roles based on performance indicators helps avoid getting stuck in local optima and leads to better overall performance.

Key Results

  • Using PAA-ResNet50 as a teacher, LAD improved PAA-ResNet101 to 46.0AP and PAA-ResNeXt101 to 47.5AP on COCO test-dev.
  • With a stronger teacher (PAA-SwinB), LAD improved PAA-ResNet50 to 43.7AP and PAA-ResNet101 to 47.9AP.
  • CoLAD, trained without a pre-trained teacher, improved baseline PAA-R50 by +0.9AP and PAA-R101 by +0.5AP in the 1x schedule.
  • CoLAD also showed improvements when one of the networks was pre-trained, with the student PAA-R101 achieving 44.1AP when trained with a PAA-R50 teacher.
  • The paper demonstrates that LAD can be combined with soft-label distillation (SoLAD), achieving a cumulative improvement of +2.0AP.
  • When using a Vision Transformer backbone (Swin-B), LAD achieved 52.0AP, approaching Cascade Mask-RCNN without segmentation training.

Limitations and Future Work

  • Teacher-Student Gap: While LAD and CoLAD show that a large gap between teacher and student performance doesn't necessarily hurt, it also means a better teacher might not provide further improvements. The authors suggest combining LAD with other distillation techniques to fully leverage a strong teacher.
  • CoLAD Complexity: While CoLAD removes the need for a pre-trained teacher, the dynamic switching mechanism and concurrent training might introduce complexity in hyperparameter tuning.
  • Future Work: The authors express optimism about LAD and CoLAD, suggesting they are topics worth further exploration and can be combined with other methods to further boost object detection performance. They also propose modifying the PAA head architecture with an auxiliary "objectness" prediction for improved results.

Summary updated

BibTeX
@inproceedings{nguyen2022improving,
  author = {Chuong H. Nguyen and Thuy C. Nguyen and Tuan N. Tang and Nam L.H. Phan},
  title = {{Improving Object Detection by Label Assignment Distillation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure