Information-theoretic regularization for learning global features by sequential VAE

Kei Akuzawa, Yusuke Iwasawa, Yutaka Matsuo

Abstract

Sequential Variational Autoencoders (VAEs) aim to disentangle global features (e.g., identity) from local features (e.g., content) in sequential data. However, standard regularization techniques often lead to "posterior collapse" or redundant information sharing between latent variables. This paper introduces a novel "CMI-maximizing" regularization objective that explicitly forces global and local features apart. By maximizing Conditional Mutual Information (CMI) through adversarial training, the proposed method achieves superior disentanglement and performance in tasks such as speaker verification and controlled image generation.

Introduction: The Challenge of Disentanglement

In representation learning, a primary goal is to separate global factors—features that remain constant across a sequence—from local factors that vary. Sequential VAEs typically employ a global latent variable $z$ and a sequence of local variables $s$ to achieve this.

Despite their design, these models suffer from two main issues:

  1. Posterior Collapse (PC): Powerful decoders may ignore the global variable $z$, rendering it uninformative.
  2. The Redundancy Trap: Common fixes for PC involve maximizing the Mutual Information (MI) between the data $x$ and $z$. However, this often forces $z$ to capture local details or allows $s$ to retain global information, leading to "entanglement."

Proposed Method: CMI-Maximizing Regularization

The authors propose a novel objective function termed CMI-maximizing regularization. The core idea is to maximize the information in $z$ conditioned on the local features $s$, denoted as $I(x; z | s)$.

The Information-Theoretic Objective

The standard MI decomposition is $I(x; z) = I(x; z | s) + I(z; s)$. Traditional regularization focuses solely on $I(x; z)$, which can inadvertently increase $I(z; s)$, the redundant information shared between global and local variables.

Comparison of standard MI-maximizing regularization vs. the proposed CMI-maximizing method.

The proposed method modifies the Evidence Lower Bound (ELBO) by maximizing $I(x; z)$ to ensure $z$ is informative, while simultaneously minimizing $I(z; s)$ to ensure $z$ and $s$ are independent.

Adversarial Estimation

Because $I(z; s)$ is difficult to compute directly, the authors utilize the Density Ratio Trick (DRT). An adversarial discriminator is trained to distinguish between true $(z, s)$ pairs and randomized pairs. The VAE then learns to "fool" this discriminator, effectively minimizing the mutual information between the two latent spaces.

Model Architectures

The method is evaluated on two distinct sequential architectures:

  1. Disentangled Sequential Autoencoders (DSAE): Often used for speech, where local variables often "leak" global information.
  2. PixelCNN-VAEs: Used for images, where powerful autoregressive decoders might cause the global variable to "hog" local details.
Graphical models for (a) State Space Models (SSM) like DSAEs and (b) Autoregressive Models (ARM) like PixelCNN-VAEs.

Experimental Evaluation

The researchers tested their approach across speech and image domains using the TIMIT corpus for audio and MNIST/Fashion-MNIST for images.

Key Results

  • Speaker Verification: CMI-VAE achieved a lower Equal Error Rate (EER) on the TIMIT dataset, indicating that $z$ captured speaker identity more precisely than baselines like $\beta$-VAE.
  • Image Classification: The global variable $z$ yielded higher classification accuracy (AoLR), even in highly compressed latent spaces.
  • Generation Quality: The model produced diverse images that remained consistent with the global category $z$, avoiding the "blurriness" typical of standard VAEs. The authors introduced a modified Classification Accuracy Score (mCAS) to quantify this consistency.

Conclusion and Future Work

This paper demonstrates that explicit independence constraints are necessary for effective disentanglement in sequential VAEs. While the CMI-maximizing approach shows significant promise, future work includes addressing hyperparameter sensitivity and improving the stability of the adversarial training process.

Summary updated

Abstract

Sequential variational autoencoders (VAEs) with a global latent variable z have been studied for disentangling the global features of data, which is useful for several downstream tasks. To further assist the sequential VAEs in obtaining meaningful z, existing approaches introduce a regularization term that maximizes the mutual information (MI) between the observation and z. However, by analyzing the sequential VAEs from the information-theoretic perspective, we claim that simply maximizing the MI encourages the latent variable to have redundant information, thereby preventing the disentanglement of global features. Based on this analysis, we derive a novel regularization method that makes z informative while encouraging disentanglement. Specifically, the proposed method removes redundant information by minimizing the MI between z and the local features by using adversarial training. In the experiments, we trained two sequential VAEs, state-space and autoregressive model variants, using speech and image datasets. The results indicate that the proposed method improves the performance of downstream classification and data generation tasks, thereby supporting our information-theoretic perspective for the learning of global features.
BibTeX
@article{akuzawa2021information,
  author = {Kei Akuzawa and Yusuke Iwasawa and Yutaka Matsuo},
  title = {{Information-theoretic regularization for learning global features by sequential VAE}},
  journal = {Machine Learning},
  year = {2021},
  doi = {10.1007/s10994-021-06032-4},
}
Figure