Lifelong Mixture of Variational Autoencoders

Fei Ye, Adrian G. Bors

TL;DR

This paper introduces the Lifelong Mixture of Variational Autoencoders (L-MVAE), a novel framework for lifelong learning that addresses catastrophic forgetting by employing a mixture of VAE "experts." The L-MVAE learns new tasks sequentially while retaining knowledge from previous ones, dynamically adapting its architecture and efficiently utilizing a single expert during inference.

What Problem Does This Paper Address?

Traditional deep learning models suffer from "catastrophic forgetting," meaning they rapidly lose previously learned information when trained on new tasks. Lifelong learning aims to overcome this by enabling models to continuously acquire, preserve, and transfer knowledge. Existing approaches often involve complex architectural changes, regularization techniques, or generative replay, which can be computationally expensive, require significant memory, or lead to performance degradation, especially with complex data or long task sequences. This paper tackles the challenge of learning new tasks without forgetting old ones, while also enabling efficient representation learning across diverse domains.

Methods

The core of the L-MVAE is a mixture of experts, where each expert is a Variational Autoencoder (VAE). The system is trained by maximizing a mixture of individual component evidence lower bounds (MELBO). A key innovation is the use of a Dirichlet distribution to dynamically learn the mixing coefficients, which control the contribution of each expert. This allows the model to adapt its architecture by activating or deactivating experts.

For training, the model uses an "assignment vector" to determine which experts are specialized for specific tasks. During inference, a selection process identifies the most relevant expert for a given data sample, ensuring computational efficiency. The L-MVAE also incorporates an "expansion mechanism" that can add new components to the mixture when a task is significantly novel, preventing the model from becoming overwhelmed. The paper also details how L-MVAE can be adapted for unsupervised, supervised, and semi-supervised learning paradigms by modifying its objective function.

Datasets

The paper evaluates L-MVAE on several standard datasets for lifelong learning tasks:

  • Supervised Learning: MNIST, Fashion-MNIST, SVHN, and CIFAR10 (referred to as the MFSC sequence).
  • Semi-supervised Learning: MNIST, Fashion-MNIST, SVHN, and CIFAR10.
  • Unsupervised Learning & Disentangled Representation Learning: CelebA, CACD, 3D-chairs, and Omniglot (referred to as the CCDO databases).
  • Transfer Metric Evaluation: MNIST, Fashion-MNIST, SVHN, and CIFAR10.
  • Expandable Mixture Model Evaluation: CIFAR10, CIFAR100, and ImageNet.

Key Insights and Ideas

  • Mixture of Experts for Lifelong Learning: Using a mixture of VAEs as experts allows for specialization, where each expert can learn a distinct latent representation for a specific task or domain.
  • Dynamic Architecture Adaptation: The Dirichlet distribution for mixing coefficients and the assignment vector mechanism enable the model to dynamically activate or deactivate experts, effectively managing knowledge and preventing forgetting.
  • Efficient Inference: By selecting a single expert during testing, L-MVAE significantly reduces computational cost compared to models that might use all components.
  • Architecture Expansion: The ability to add new components when encountering novel tasks is crucial for handling an ever-increasing number of diverse learning challenges.
  • Cross-Domain Representation: L-MVAE excels at learning representations that span multiple data domains, a significant advantage over models trained on single domains.

Key Results

  • Supervised Learning: L-MVAE demonstrated strong classification accuracy across the MFSC sequence (MNIST, Fashion, SVHN, CIFAR10), maintaining performance on previously learned tasks and outperforming methods like Deep Generative Replay (DGR). Table I shows L-MVAE achieving accuracies like 97.97% on MNIST and 69.32% on CIFAR10.
  • Semi-supervised Learning: The model achieved competitive results compared to state-of-the-art methods, even when trained on a single dataset. Table II shows classification errors on MNIST, with L-MVAE achieving 4.95.
  • Unsupervised Learning: L-MVAE successfully performed image reconstruction and latent space interpolation across diverse domains (CelebA, CACD, 3D-chairs, Omniglot), showcasing its ability to capture underlying data structures. Figures 4 and 5 illustrate impressive reconstruction quality.
  • Disentangled Representation Learning: The model discovered disentangled representations for attributes like age, hair style, and illumination in CelebA images (Figure 7).
  • Transfer Learning: L-MVAE showed quick convergence when learning new tasks, indicating strong transfer learning capabilities (Figure 11).

Limitations and Future Work

  • Number of Components: The paper notes that the number of components doesn't drastically affect performance, and they typically use K=4, suggesting that a fixed number might not be optimal for all scenarios. Future work could explore more adaptive ways to determine the optimal number of components.
  • Dirichlet Parameter Estimation: The study mentions an ablation where Dirichlet parameters were not properly estimated, leading to performance degradation. This highlights the importance of accurate parameter estimation for the model's effectiveness.
  • Over-regularization: The paper discusses the potential for over-regularization in VAEs and proposes using a B-VAE approach with a tunable parameter $\beta$ to mitigate this. Further investigation into optimal $\beta$ scheduling could be a future direction.
  • Computational Cost: While inference is efficient, training a mixture model with many components can still be computationally intensive.
  • Dataset Order Impact: While the paper suggests the order of learning databases has minimal impact on L-MVAE training (Table VII), further exploration of this could be beneficial.

Summary updated

Abstract

In this paper, we propose an end-to-end lifelong learning mixture of experts. Each expert is implemented by a Variational Autoencoder (VAE). The experts in the mixture system are jointly trained by maximizing a mixture of individual component evidence lower bounds (MELBO) on the log-likelihood of the given training samples. The mixing coefficients in the mixture, control the contributions of each expert in the goal representation. These are sampled from a Dirichlet distribution whose parameters are determined through non-parametric estimation during lifelong learning. The model can learn new tasks fast when these are similar to those previously learnt. The proposed Lifelong mixture of VAE (L-MVAE) expands its architecture with new components when learning a completely new task. After the training, our model can automatically determine the relevant expert to be used when fed with new data samples. This mechanism benefits both the memory efficiency and the required computational cost as only one expert is used during the inference. The L-MVAE inference model is able to perform interpolation in the joint latent space across the data domains associated with different tasks and is shown to be efficient for disentangled learning representation.
BibTeX
@article{ye2021lifelong,
  author = {Fei Ye and Adrian G. Bors},
  title = {{Lifelong Mixture of Variational Autoencoders}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2021},
  doi = {10.1109/tnnls.2021.3096457},
}
Figure