Self-training and multi-task learning for limited data: evaluation study on object detection

Hoàng-Ân Lê, Minh-Tan PhamWorkshops @ ICCV 2023 · DOI · arXiv

Introduction

Deep learning models are notoriously data-hungry, and object detection remains one of the most expensive tasks to annotate. While techniques like self-training (where a "teacher" model labels unlabeled data for a "student") and multi-task learning (MTL, where a model learns multiple tasks simultaneously) are common, they usually rely on heavy assumptions. Standard self-training assumes the teacher is highly accurate and has already seen the student's training data, while standard MTL assumes every image is annotated for every task.

This paper investigates the "deficiency scenario," addressing three critical questions:

  1. Can a "weak" teacher (trained on minimal data) still guide a student effectively?
  2. Can self-training succeed when teacher and student training sets are entirely disjoint?
  3. How can we leverage different datasets for different tasks (e.g., detection vs. segmentation) when no images share labels for both?

Methodology

The authors evaluate a teacher-student framework using a ResNet50 teacher and a smaller ResNet18 student. The approach rests on three technical pillars:

  • Self-Training with Pseudo-labels: The teacher generates "soft" labels (predictions) on unlabeled data, which the student uses for training.
  • Feature-Imitation Knowledge Distillation (KD): The student mimics the teacher's internal feature maps. Three variants were tested: MSE (simple mean square error), PDF-Distil (focusing on teacher-student disagreement), and DeFeat (distilling foreground and background separately).
  • Partially Annotated Multi-Task Learning: A single network is trained for both object detection and semantic segmentation using a "disjoint" setup. Each image is labeled for only one task, and an alternating optimization strategy updates the shared encoder.

Experiments were conducted on the Pascal VOC and VEDAI (aerial vehicle detection) datasets, with the Semantic Boundaries Dataset (SBD) providing additional segmentation labels.

Key Results and Insights

The Power of Soft Labels and Distillation

A primary finding is that a student trained on a large set of "noisy" pseudo-labels from a weak teacher often outperforms a student trained on a small set of perfect, human-annotated ground truths. On Pascal VOC, a supervised student trained on 50% of the data achieved 51.08 mAP. By utilizing a teacher to label the remaining 50%, performance rose to 55.64 mAP. Adding PDF-Distil further boosted this to 56.47 mAP, nearly closing the gap with the larger teacher model (58.24 mAP).

Performance of self-trained models vs. supervised baselines on VOC and VEDAI datasets.

Cross-Task Synergy in MTL

In the multi-task scenario with disjoint labels, the student reached 47.77 mAP. This significantly outperforms the single-task supervised baseline (40.45 mAP) without requiring any additional detection labels—effectively "borrowing" features from the segmentation task. This suggests that the underlying feature representations for "what" an object is (segmentation) are highly beneficial for "where" an object is (detection), even across different task topologies.

Multi-task learning results for detection and segmentation with disjoint labels.

Generalization and Thresholds

The study reveals that a teacher does not need to have seen the student's data to be an effective mentor; knowledge generalizes well even across separate training sets. However, there is a "toleration threshold": while performance is stable when reducing teacher data from 50% to 25%, dropping to 12.5% leads to a performance cliff as predictions become too unreliable.

Conclusion and Future Work

The research demonstrates that self-training and multi-task learning can significantly mitigate data scarcity in object detection. While the student is ultimately bound by the teacher's performance, the use of feature distillation and cross-task knowledge transfer provides a robust path forward for low-data regimes. Future work will explore the "domain gap"—evaluating these methods when the teacher and student are trained on different types of imagery, such as natural versus satellite data.

Summary updated

BibTeX
@inproceedings{le2023self,
  author = {Hoàng-Ân Lê and Minh-Tan Pham},
  title = {{Self-training and multi-task learning for limited data: evaluation study on object detection}},
  booktitle = {Workshops @ ICCV},
  year = {2023},
  doi = {10.1109/iccvw60793.2023.00107},
}
Figure