Wasserstein Expansible Variational Autoencoder for Discriminative and Generative Continual Learning

Fei Ye, Adrian G. Bors

Abstract

The Wasserstein Expansible Variational Autoencoder (WEVAE) is a novel framework for Task-Free Continual Learning (TFCL) that dynamically grows its architecture only when it encounters statistically novel data. By using Wasserstein distance as an expansion criterion and an energy-based sampling method for memory management, it achieves state-of-the-art performance in both generating and classifying data while maintaining a significantly more compact model than its predecessors.

Introduction

Continual Learning (CL) aims to train models on a stream of data without forgetting what was learned previously—a phenomenon known as catastrophic forgetting. Most CL methods rely on "task boundaries" (knowing exactly when the data distribution shifts, e.g., moving from recognizing birds to recognizing cars).

However, real-world data is often Task-Free (TFCL), meaning it’s a continuous, non-stationary stream without clear labels for when a new "task" starts. Existing mixture-of-experts approaches (like mixture VAEs) often suffer from two issues: redundancy, where they expand the model too frequently, and a lack of theoretical rigor in their expansion mechanisms. WEVAE addresses these by focusing on knowledge diversity and statistical distance.

Methods

The authors introduce two primary mechanisms to make the VAE mixture more efficient and robust:

  1. Wasserstein Expansion Mechanism: Instead of expanding the model based on simple loss spikes, WEVAE evaluates the statistical similarity between the current component and all previously learned components. It uses the Wasserstein distance (or Earth Mover's Distance) to measure this. If the new data represents a distribution that is sufficiently "far" from what the model already knows—tracked over a time-series window to avoid reacting to outliers—a new VAE component is added.
  2. Energy-Based Sample Selection: To manage a finite memory buffer, the model assigns an "energy" score to samples based on their reconstruction error across all existing components. High energy indicates a "novel" sample that the model hasn't mastered yet, so it stays in the buffer. Low energy indicates "redundant" data that the model already understands, which can be safely discarded.
  3. Unified Architecture: The model is designed for both generative tasks (reconstructing images) and discriminative tasks (classification) by treating class labels as latent variables within a probabilistic framework.
Figure 1: Overview of the WEVAE framework, illustrating the sample selection process via energy functions and the expansion checking mechanism using Wasserstein distance.

Key Results and Insights

The model was evaluated on several standard and complex benchmarks, including Split MNIST, Split CIFAR10, and Split CIFAR100, as well as cross-domain streams like MNIST + SVHN.

  • Superior Density Estimation: On the Split MNIST-Fashion task, WEVAE achieved a log-likelihood of -172.47, significantly outperforming the previous state-of-the-art, Dynamic-ELBO-OCM (-187.49).
  • Extreme Compactness: WEVAE often achieves better results with far fewer components. For example, on Split MNIST, it used only 3 components to outperform models using 30.
  • State-of-the-Art Classification: On Split CIFAR100, WEVAE reached 25.07% accuracy, beating other dynamic expansion models like CNDPM (20.10%) and Dynamic-OCM (21.79%).
Figure 6: Sensitivity analysis and model expansion dynamics. (a) Impact of memory size on performance. (b) Effect of varying the expansion threshold λ on the number of experts. (c) Influence of batch size. (d) Visualization of model expansion (number of experts) relative to training steps and distribution IDs.

Key Insights

  • Diversity over Quantity: The core innovation is that model expansion should be driven by knowledge diversity. By forcing new components to be statistically distinct from old ones via the Wasserstein metric, the authors ensure each "expert" in the mixture specializes in a unique part of the data manifold.
  • Time-Series Smoothing: Expansion isn't triggered by a single bad batch. By treating the similarity scores as a stochastic process, the model becomes resilient to noise and "bursty" data.

Limitations and Future Work

While WEVAE shows significant improvements, the expansion threshold (λ) remains a critical hyperparameter that requires careful tuning for unknown data streams. Additionally, while the model is parameter-efficient, managing a mixture of VAEs and calculating Wasserstein distances introduces computational overhead. Future research will focus on scaling the expansion logic to high-resolution, complex natural images like ImageNet.

Summary updated

Abstract

Task-Free Continual Learning (TFCL) represents a challenging learning paradigm where a model is trained on the non-stationary data distributions without any knowledge of the task information, thus representing a more practical approach. Despite promising achievements by the Variational Autoencoder (VAE) mixtures in continual learning, such methods ignore the redundancy among the probabilistic representations of their components when performing model expansion, leading to mixture components learning similar tasks. This paper proposes the Wasserstein Expansible Variational Autoencoder (WEVAE), which evaluates the statistical similarity between the probabilistic representation of new data and that represented by each mixture component and then uses it for deciding when to expand the model. Such a mechanism can avoid unnecessary model expansion while ensuring the knowledge diversity among the trained components. In addition, we propose an energy-based sample selection approach that assigns high energies to novel samples and low energies to the samples which are similar to the model’s knowledge. Extensive empirical studies on both supervised and unsupervised benchmark tasks demonstrate that our model outperforms all competing methods. The code is available at https://github.com/dtuzi123/WEVAE/.
BibTeX
@inproceedings{ye2023wasserstein,
  author = {Fei Ye and Adrian G. Bors},
  title = {{Wasserstein Expansible Variational Autoencoder for Discriminative and Generative Continual Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01711},
}
Figure