Continual Learning of Generative Models With Limited Data: From Wasserstein-1 Barycenter to Adaptive Coalescence

Mehmet Dedeoglu, Sen Lin, Zhaofeng Zhang, Junshan Zhang

TL;DR

This paper tackles the challenge of training generative models on resource-constrained edge devices with limited local data. They propose a novel framework that leverages pre-trained models from other edge nodes by treating them as "knowledge balls" and coalescing them using Wasserstein-1 barycenters. This approach effectively mitigates catastrophic forgetting and enables efficient continual learning for generative models at the edge.

What Problem Does This Paper Address?

The proliferation of Internet of Things (IoT) devices has created a demand for AI capabilities at the network edge. However, training deep generative models, which are crucial for tasks like data synthesis and privacy preservation, is computationally intensive and data-hungry. Traditional approaches involve sending data to the cloud for training, which incurs significant communication overhead and raises privacy concerns. While transferring knowledge from pre-trained models is a promising avenue, existing methods like simple weight averaging or direct initialization often suffer from catastrophic forgetting (losing previously learned information) or require substantial computational resources. This paper aims to bridge this gap by developing a method for continual learning of generative models at the edge that efficiently utilizes knowledge from multiple pre-trained models while minimizing data and computational requirements.

Methods

The core of the proposed solution lies in framing continual learning as an adaptive coalescence problem, solved using Wasserstein-1 barycenters.

  • Wasserstein-1 Barycenter Formulation: The authors treat the knowledge from pre-trained generative models at other edge nodes as "Wasserstein balls" centered around their respective models. The goal is to find a new generative model that is a "barycenter" of these pre-trained models and the local data distribution. This is formulated as a constrained optimization problem.
  • Two-Stage Recursive Algorithm: To efficiently solve this barycenter problem, a two-stage approach is employed:
  • Offline Barycenter Computation (Cloud): A "recursive" Wasserstein Generative Adversarial Network (WGAN) configuration with two discriminators is used to compute the barycenter of the pre-trained models. This process is performed offline in the cloud.
  • Fast Adaptation (Edge): The barycenter computed offline serves as a meta-model initialization for the target edge node. Fast adaptation is then performed using local data samples to fine-tune the generative model.
  • Recursive WGAN: A key implementation detail is the use of a recursive WGAN configuration. At each recursive step, a 2-discriminator WGAN is trained to find the barycenter between the current barycenter and the next pre-trained model. This approach avoids the need for a single, massive multi-discriminator GAN.
  • Weight Ternarization: To further compress the generative model for efficient edge deployment, a weight ternarization method is employed. This technique jointly optimizes weights and thresholds for quantization, reducing computational cost and model size with minimal performance degradation.

Datasets

The authors evaluate their framework on several standard datasets:

  • CIFAR10
  • CIFAR100
  • LSUN
  • MNIST

Key Insights and Ideas

  • Knowledge as "Wasserstein Balls": Representing knowledge from pre-trained models as Wasserstein balls provides a principled way to quantify and leverage their relevance for continual learning.
  • Wasserstein Barycenters for Model Fusion: Using Wasserstein-1 barycenters offers a robust mathematical framework for fusing knowledge from multiple generative models, overcoming limitations of simpler averaging methods.
  • Recursive WGAN for Scalability: The recursive WGAN architecture is crucial for making the barycenter computation tractable, especially when dealing with a large number of pre-trained models.
  • Generative Replay for Catastrophic Forgetting: The proposed barycentric approach implicitly incorporates generative replay, which is shown to be effective in mitigating catastrophic forgetting.
  • Weight Ternarization for Edge Efficiency: The integration of weight ternarization demonstrates a practical approach to deploying these models on resource-constrained edge devices.

Key Results

  • The proposed barycentric fast adaptation framework consistently outperforms baseline methods like "Edge-Only" (training solely on local data) and "Transferring GANs" (using pre-trained models as initialization) across various datasets (MNIST, CIFAR10, CIFAR100, LSUN).
  • The method achieves better image quality (lower FID scores) and faster convergence, especially in scenarios with limited local data.
  • The impact of Wasserstein ball radii (ηk) is demonstrated: smaller radii (higher relevance) lead to better performance, and the framework can adapt by assigning higher weights to more relevant models.
  • Weight ternarization successfully compresses the model, enabling efficient edge learning with negligible performance degradation compared to full-precision models.
  • The framework shows promise in handling both overlapping and non-overlapping class distributions between source and target edge nodes.

Limitations and Future Work

  • Non-uniqueness of Barycenters: The paper acknowledges that Wasserstein-1 barycenters can be non-unique, which can influence the final model. The proposed recursive approach aims to navigate this by considering the order of coalescence.
  • Computational Cost of Offline Training: While the recursive approach reduces complexity, the offline barycenter computation in the cloud can still be computationally intensive, especially with a very large number of source nodes.
  • Future Work: The authors suggest exploring the impact of different distance functionals beyond Wasserstein-1 and investigating more advanced techniques for adaptive weight assignment based on model relevance. They also hint at the potential for this framework in applications like image morphing, clustering, and privacy-aware synthetic data generation.

Summary updated

Abstract

Learning generative models is challenging for a network edge node with limited data and computing power. Since tasks in similar environments share a model similarity, it is plausible to leverage pretrained generative models from other edge nodes. Appealing to optimal transport theory tailored toward Wasserstein-1 generative adversarial networks (WGANs), this study aims to develop a framework that systematically optimizes continual learning of generative models using local data at the edge node while exploiting adaptive coalescence of pretrained generative models. Specifically, by treating the knowledge transfer from other nodes as Wasserstein balls centered around their pretrained models, continual learning of generative models is cast as a constrained optimization problem, which is further reduced to a Wasserstein-1 barycenter problem. A two-stage approach is devised accordingly: 1) the barycenters among the pretrained models are computed offline, where displacement interpolation is used as the theoretic foundation for finding adaptive barycenters via a "recursive" WGAN configuration and 2) the barycenter computed offline is used as metamodel initialization for continual learning, and then, fast adaptation is carried out to find the generative model using the local samples at the target edge node. Finally, a weight ternarization method, based on joint optimization of weights and threshold for quantization, is developed to compress the generative model further. Extensive experimental studies corroborate the effectiveness of the proposed framework.
BibTeX
@article{dedeoglu2021continual,
  author = {Mehmet Dedeoglu and Sen Lin and Zhaofeng Zhang and Junshan Zhang},
  title = {{Continual Learning of Generative Models With Limited Data: From Wasserstein-1 Barycenter to Adaptive Coalescence}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2021},
  doi = {10.1109/tnnls.2023.3251096},
}
Figure