Evolve: Enhancing Unsupervised Continual Learning with Multiple Experts

Xiaofan Yu, Tajana Rosing, Yunhui GuoWACV 2024 · DOI

Introduction

Unsupervised Continual Learning (UCL) represents a significant frontier for autonomous systems, enabling agents to learn from raw, unlabeled data streams. However, real-world data is typically non-i.i.d., characterized by temporal correlations where objects appear sequentially. Standard Self-Supervised Learning (SSL) methods, designed for i.i.d. data, suffer from catastrophic forgetting in these scenarios, often performing worse than randomly initialized models as they overfit to current temporal features.

The EVOLVE Framework

To address these challenges, we propose EVOLVE, a hybrid cloud-client framework that leverages diverse, pre-trained "expert" models to guide local learning.

Detailed architecture of the EVOLVE framework showing the interaction between the cloud-based expert models and the client-side target model.

Hybrid Architecture and Expert Aggregation

EVOLVE utilizes a small target model on the client device and a library of large, pre-trained expert models (e.g., Swin Transformers, ResNets) in the cloud. Knowledge transfer is achieved via the Expert Aggregation Loss ($\mathcal{L}_E$), which employs the Hilbert-Schmidt Independence Criterion (HSIC) to align the local model's representations with those of the experts.

Dynamic Weight Adjustment

Recognizing that expert relevance varies by context, EVOLVE implements a dynamic weighting mechanism. A Confidence Metric is calculated for each expert based on its predictive consistency across different views of the same data. These weights are updated using a moving average, allowing the system to adapt to environmental shifts without prematurely discarding potentially useful experts.

Experimental Evaluation

EVOLVE was evaluated on benchmarks including CIFAR-10, TinyImageNet, CORe50, and Stream-51, specifically focusing on sequential-imbalanced (Seq-imb) setups that mimic real-world distribution shifts.

Results and Discussion

EVOLVE consistently outperformed existing UCL methods across all benchmarks.

Comparison of EVOLVE against state-of-the-art UCL methods across different SSL backbones, demonstrating superior accuracy and stability.

Key findings include:

  • Accuracy Gains: EVOLVE improved linear evaluation accuracy by 6.1% to 53.7% over state-of-the-art methods. On the challenging Stream-51 dataset, EVOLVE achieved over 84% accuracy when paired with a BYOL backbone, while standard SSL methods struggled to exceed 30%.
  • Stability: While competing methods often show "sawtooth" performance due to forgetting, EVOLVE maintains steady, monotonic improvement.
  • Efficiency: The framework is practical for real-time use, with communication overhead under 1ms over 5G networks (roughly 720kB per round).

Conclusion and Future Work

EVOLVE effectively mitigates catastrophic forgetting in UCL by using pre-trained experts as visual anchors. Future work will address privacy through techniques like Fully Homomorphic Encryption (FHE) and explore methods to reduce dependency on high-speed communication and specific expert libraries.

Summary updated

BibTeX
@inproceedings{yu2024evolve,
  author = {Xiaofan Yu and Tajana Rosing and Yunhui Guo},
  title = {{Evolve: Enhancing Unsupervised Continual Learning with Multiple Experts}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00236},
}
Figure