Incremental Learning with Unlabeled Data in the Wild.

Kibok Lee, Kimin Lee, Jinwoo Shin, Honglak LeeWorkshops @ CVPR 2019 · OpenReview

Introduction

Deep Neural Networks (DNNs) excel at learning from static datasets but struggle with "lifelong learning." When a model is trained on a sequence of new tasks—such as learning to recognize dogs, then birds, then cars—it tends to abruptly forget previously acquired knowledge. This phenomenon, known as catastrophic forgetting, remains a significant hurdle in deploying AI systems in dynamic environments.

Standard approaches to Class-Incremental Learning (CIL) typically involve maintaining a "coreset" of old data to refresh the model’s memory. However, memory is often a constrained resource. This paper proposes a paradigm shift: instead of relying solely on limited labeled data, models can leverage the virtually infinite stream of unlabeled data available "in the wild" (e.g., social media or web scrapes) to maintain their internal representations of previous tasks.

The Proposed Framework

The authors introduce a novel framework designed to integrate unlabeled data into the incremental learning process through three primary innovations: Global Distillation, a structured 3-step learning scheme, and confidence-based sampling.

1. Global Distillation (GD)

Traditional CIL methods often utilize "local distillation," which helps a model remember classes within a specific task. However, this approach frequently fails to help the model distinguish between classes across different tasks (inter-task discrimination).

Global Distillation addresses this by employing an ensemble "teacher" model. This teacher combines the knowledge of the previous model with a new teacher specialized for the current task. By distilling knowledge from this unified ensemble, the student model is forced to learn the decision boundaries between all classes encountered throughout its training history.

2. 3-Step Learning Scheme

To effectively incorporate new information without biasing the model, the authors implement a three-stage training process:

  • Step 1: Teacher Specialization. A temporary teacher model is trained exclusively on the new labeled task data.
  • Step 2: Global Distillation. The main model is trained using the GD loss, pulling knowledge from both the previous iteration of the model and the new specialized teacher.
  • Step 3: Balanced Fine-tuning. Because new tasks typically have significantly more data than the old coresets, models often develop a bias toward new classes. The authors utilize a data-weighting technique to rebalance gradients, effectively eliminating this bias.

3. Confidence-Based Sampling

Not all unlabeled data is beneficial. The framework includes a sampling algorithm to identify "wild" data that the model is already confident about, as these samples likely belong to previously learned classes. To prevent the model from becoming overconfident on irrelevant noise, the authors also incorporate "Out-of-Distribution" (OOD) data. This calibration step ensures the model remains "honest" about its recognition capabilities.

Experimental Evaluation

The researchers evaluated their method using standard benchmarks (CIFAR-100 and ImageNet) alongside massive unlabeled streams: TinyImages (80 million images) and ImageNet 2011 (14 million images). To ensure the integrity of the results, any classes present in the labeled benchmarks were strictly excluded from the unlabeled streams.

Key Results

The proposed method demonstrated substantial improvements over previous state-of-the-art (SOTA) methods like iCaRL and LwF:

  • Accuracy Gains: On a 5-task ImageNet split, the method achieved 55.2% accuracy, a significant jump from the 47.7% achieved by the previous best method (E2E).
  • Forgetting Reduction: The "Average Forgetting" (FGT) metric was reduced by 46.5% (dropping from 17.9% to 9.6%) when external unlabeled data was utilized.
  • Scalability: The performance gains remained consistent across various task granularities (5, 10, or 20 classes per task). Notably, even without external data, the Global Distillation and fine-tuning components alone outperformed existing SOTA techniques.

Discussion and Limitations

While the results are promising, the authors acknowledge certain trade-offs. The confidence-based sampling process is computationally intensive, requiring millions of images to pass through the model. Additionally, the method assumes the availability of a large stream of unlabeled data, which may not be present in specialized fields like medical imaging.

Future research may focus on more sophisticated methods for correlating unlabeled data with labeled tasks, perhaps by utilizing semantic hierarchies to select even more relevant samples from the wild.

Conclusion

By treating unlabeled data as a "structural anchor" rather than a direct source of labels, this research provides a robust path forward for lifelong learning. The combination of Global Distillation and strategic sampling allows DNNs to combat catastrophic forgetting more effectively than ever before, paving the way for more resilient and adaptive AI systems.

Summary updated

Abstract

We propose to leverage a continuous and large stream of unlabeled data in the wild to alleviate catastrophic forget- ting in class-incremental learning. Our experimental results on CIFAR and ImageNet datasets demonstrate the superiority of the proposed methods over prior methods: compared to the state-of-the-art method, our proposed method shows up to 14.9% higher accuracy and 45.9% less forgetting.
BibTeX
@inproceedings{lee2019incremental,
  author = {Kibok Lee and Kimin Lee and Jinwoo Shin and Honglak Lee},
  title = {{Incremental Learning with Unlabeled Data in the Wild.}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=Bixx4TpfeOaB},
}

Loading related content...

Figure