Abstract
Sequential Variational Autoencoders (VAEs) aim to disentangle global features (e.g., identity) from local features (e.g., content) in sequential data. However, standard regularization techniques often lead to "posterior collapse" or redundant information sharing between latent variables. This paper introduces a novel "CMI-maximizing" regularization objective that explicitly forces global and local features apart. By maximizing Conditional Mutual Information (CMI) through adversarial training, the proposed method achieves superior disentanglement and performance in tasks such as speaker verification and controlled image generation.
Introduction: The Challenge of Disentanglement
In representation learning, a primary goal is to separate global factors—features that remain constant across a sequence—from local factors that vary. Sequential VAEs typically employ a global latent variable $z$ and a sequence of local variables $s$ to achieve this.
Despite their design, these models suffer from two main issues:
- Posterior Collapse (PC): Powerful decoders may ignore the global variable $z$, rendering it uninformative.
- The Redundancy Trap: Common fixes for PC involve maximizing the Mutual Information (MI) between the data $x$ and $z$. However, this often forces $z$ to capture local details or allows $s$ to retain global information, leading to "entanglement."
Proposed Method: CMI-Maximizing Regularization
The authors propose a novel objective function termed CMI-maximizing regularization. The core idea is to maximize the information in $z$ conditioned on the local features $s$, denoted as $I(x; z | s)$.
The Information-Theoretic Objective
The standard MI decomposition is $I(x; z) = I(x; z | s) + I(z; s)$. Traditional regularization focuses solely on $I(x; z)$, which can inadvertently increase $I(z; s)$, the redundant information shared between global and local variables.
The proposed method modifies the Evidence Lower Bound (ELBO) by maximizing $I(x; z)$ to ensure $z$ is informative, while simultaneously minimizing $I(z; s)$ to ensure $z$ and $s$ are independent.
Adversarial Estimation
Because $I(z; s)$ is difficult to compute directly, the authors utilize the Density Ratio Trick (DRT). An adversarial discriminator is trained to distinguish between true $(z, s)$ pairs and randomized pairs. The VAE then learns to "fool" this discriminator, effectively minimizing the mutual information between the two latent spaces.
Model Architectures
The method is evaluated on two distinct sequential architectures:
- Disentangled Sequential Autoencoders (DSAE): Often used for speech, where local variables often "leak" global information.
- PixelCNN-VAEs: Used for images, where powerful autoregressive decoders might cause the global variable to "hog" local details.
Experimental Evaluation
The researchers tested their approach across speech and image domains using the TIMIT corpus for audio and MNIST/Fashion-MNIST for images.
Key Results
- Speaker Verification: CMI-VAE achieved a lower Equal Error Rate (EER) on the TIMIT dataset, indicating that $z$ captured speaker identity more precisely than baselines like $\beta$-VAE.
- Image Classification: The global variable $z$ yielded higher classification accuracy (AoLR), even in highly compressed latent spaces.
- Generation Quality: The model produced diverse images that remained consistent with the global category $z$, avoiding the "blurriness" typical of standard VAEs. The authors introduced a modified Classification Accuracy Score (mCAS) to quantify this consistency.
Conclusion and Future Work
This paper demonstrates that explicit independence constraints are necessary for effective disentanglement in sequential VAEs. While the CMI-maximizing approach shows significant promise, future work includes addressing hyperparameter sensitivity and improving the stability of the adversarial training process.
Summary updated