Abstract
Despite recent successes in synthesizing faces and bedrooms, existing generative models struggle to capture more complex image types, potentially due to the oversimplification of their latent space constructions. To tackle this issue, building on Variational Autoencoders (VAEs), we integrate recurrent connections across channels to both inference and generation steps, allowing the high-level features to be captured in global-to-local, coarse-to-fine manners. Combined with adversarial loss, our channel-recurrent VAE-GAN (crVAE-GAN) outperforms VAE-GAN in generating a diverse spectrum of high resolution images while maintaining the same level of computational efficacy. Our model produces interpretable and expressive latent representations to benefit downstream tasks such as image completion. Moreover, we propose two novel regularizations, namely the KL objective weighting scheme over time steps and mutual information maximization between transformed latent variables and the outputs, to enhance the training.
Introduction: The Challenge of Complex Image Synthesis
Generative modeling has long been characterized by a trade-off between Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs). While VAEs offer stable training and a structured latent space, they often produce blurry outputs. Conversely, GANs generate sharp images but are prone to "mode collapse" and training instability.
Existing hybrid models, such as the VAE-GAN, struggle with contextually complex images—for instance, a bird in a natural, cluttered habitat. The authors identify the bottleneck as the rigid fully-connected (FC) layers typically used to bridge the latent space and convolutional layers. These layers often fail to capture the nuanced local descriptions required for high-fidelity image modeling.
Methodology: The crVAE Architecture
The core innovation of this work is the Channel-Recurrent Variational Autoencoder (crVAE). Instead of utilizing a single flat vector, the model treats the latent space as a sequence of convolutional feature blocks processed via an LSTM.
- Channel-Recurrent Architecture: By slicing convolutional features across channels and feeding them into an LSTM, the model processes latent information sequentially. This enables a "coarse-to-fine" generation process: early time steps focus on global structure (outlines and background), while later steps refine local details like texture and facial expressions.
- crVAE-GAN Hybrid: To maintain sharpness, the crVAE framework is augmented with an adversarial loss through a Discriminator.
- Novel Regularization Techniques:
- KL Objective Weighting: A scheduling scheme applies higher weights to earlier time steps to enforce a tight information bottleneck for global structure, while lower weights at later steps allow for detailed flexibility.
- Mutual Information (MI) Maximization: To prevent mode collapse, the authors maximize the mutual information between latent variables and the generated output, ensuring the generator effectively utilizes the latent code.
Experimental Results
The model was evaluated on diverse datasets, including CelebA, LSUN Bedrooms, and a complex Birds dataset (combining Birdsnap, NABirds, and CUB-200-2011).
On the challenging Birds dataset (64x64), crVAE-GAN achieved an Inception Score of 11.07, nearly doubling the baseline VAE-GAN's score of 5.81. The model also excels in high-resolution synthesis; by stacking a second stage, it successfully generated 128x128 and 224x224 images. Human evaluation studies further validated the model's performance, with users preferring crVAE-GAN's image completions 70-80% of the time over baselines.
Key Insights and Efficiency
- Global-to-Local Progression: Visualizing the LSTM time steps reveals a natural hierarchy where the model "sketches" the image before "painting" the details.
- Parameter Efficiency: Despite the added complexity of the LSTM, weight sharing makes the model smaller. The crVAE-GAN contains 130M parameters, compared to 164M in a standard VAE-GAN baseline.
- Computational Speed: The model remains highly efficient, generating images in approximately 4ms—orders of magnitude faster than autoregressive models like PixelCNN.
Conclusion and Future Work
The crVAE-GAN demonstrates that channel recurrency can significantly enhance the representational power of generative models. While the latent space is more interpretable, future work may focus on making these semantic factors even more explicit. Additionally, combining channel recurrency with techniques like progressive growing could push the boundaries of high-resolution synthesis even further.
Summary updated