Channel-Recurrent Autoencoding for Image Modeling

Wenling Shang, Kihyuk Sohn, Yuandong TianWACV 2018 · OpenReview

TL;DR

This paper introduces a novel channel-recurrent Variational Autoencoder (VAE) architecture, crVAE-GAN, that significantly improves image generation quality, especially for complex datasets like birds. By incorporating recurrent connections across convolutional channels, the model captures global-to-local and coarse-to-fine image structures, leading to more realistic and diverse outputs compared to existing VAE-GAN models.

What Problem Does This Paper Address?

Generating high-quality, diverse images, especially for complex datasets, remains a significant challenge in deep learning. While models like Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) have shown promise, they often struggle with:

  • Blurry Generations: VAEs tend to produce blurry images due to their reconstruction objective and KL divergence penalty, which can over-smooth the latent space.
  • Limited Mode Coverage: GANs can be unstable and may only model high-density regions of the data distribution, failing to capture the full diversity.
  • Difficulty with Complex Data: Existing models often perform well on simpler datasets like faces or bedrooms but struggle with more intricate images, such as birds in natural habitats, due to limitations in their latent space construction.
  • Lack of Expressiveness: The latent spaces of current models may not be expressive enough to disentangle complex factors of variation in images.

This paper aims to address these limitations by proposing a more expressive and robust generative model capable of handling complex image distributions.

Methods

The core innovation of this paper is the channel-recurrent autoencoder (crVAE). This architecture builds upon the Variational Autoencoder framework but introduces a novel way to process convolutional features:

  • Recurrent Connections Across Channels: Instead of treating convolutional features independently, the crVAE uses a Long Short-Term Memory (LSTM) network to process groups of convolutional features sliced across channels sequentially. This allows information to propagate through time steps, enabling the model to capture temporal coherence and dependencies between channels.
  • Global-to-Local, Coarse-to-Fine Generation: This recurrent processing enables the model to learn global structures in earlier time steps and refine them with local details in later steps, mimicking a coarse-to-fine generation process.
  • VAE-GAN Extension: The proposed crVAE is further extended to crVAE-GAN by incorporating an adversarial loss, similar to VAE-GAN, to enhance the realism of generated images.
  • Novel Regularizations: Two new regularization techniques are introduced:
  • KL Objective Weighting: A time-dependent weighting scheme for the KL divergence term, where earlier time steps have higher weights to enforce tighter information bottlenecks for general outlines, and later steps have lower weights for finer details.
  • Mutual Information (MI) Maximization: Borrowing from existing work, MI maximization is used to encourage the latent variables to carry meaningful information about the generated output, improving training stability and semantic disentanglement.

Datasets

The paper evaluates its model on three diverse datasets:

  • Birds: A combination of Birdsnap, NABirds, and Caltech-UCSD Birds-200-2011 datasets, containing a large number of bird images.
  • CelebA: A dataset of celebrity faces.
  • LSUN Bedroom: A dataset of bedroom images.

The models are trained and evaluated on resolutions of 64x64, 128x128, and 224x224.

Key Insights and Ideas

  • Latent Space Structure Matters: The paper strongly suggests that the way latent variables are structured and processed is crucial for modeling complex image distributions. Simply using fully-connected layers between convolutional layers (as in standard VAEs or even VAE-GANs) can lose essential local information.
  • Recurrence for Temporal Coherence: Applying recurrent mechanisms (like LSTMs) across channels, rather than just spatially or temporally within a single channel, allows for a more holistic understanding of image features and their relationships.
  • Coarse-to-Fine Generation is Effective: The channel-recurrent approach naturally lends itself to a coarse-to-fine generation strategy, where broad strokes are laid down first, followed by finer details, leading to more coherent and realistic images.
  • Regularization is Key for Stability and Semantics: The proposed KL weighting and MI maximization regularizations are vital for stabilizing training, preventing mode collapse, and improving the semantic interpretability of the latent space, especially when combined with the channel-recurrent architecture.
  • MI Regularization's Compatibility: The paper highlights that MI regularization, while beneficial for crVAE-GAN, can be detrimental for standard VAE-GAN, suggesting that the channel-recurrent structure is essential for effectively leveraging such regularizations.

Key Results

  • Superior Image Generation Quality: crVAE-GAN consistently outperforms VAE-GAN in generating higher-quality, more realistic, and diverse images across all tested datasets, particularly for the challenging bird dataset.
  • Improved Inception Scores: Quantitative evaluations using inception scores show significant improvements for crVAE-GAN over VAE-GAN.
  • Higher Resolution Synthesis: The model successfully generates higher-resolution images (128x128 and 224x224) by stacking generation networks, with the quality of the first stage generation being critical.
  • Effective Image Completion: The learned latent manifold of crVAE-GAN is shown to be more semantically meaningful, leading to better performance in image completion tasks. Human evaluations confirm crVAE-GAN's superiority in filling missing image regions.
  • Computational Efficiency: Despite the added complexity of recurrent connections, crVAE-GAN maintains computational efficiency comparable to VAE-GAN, with fewer parameters and lower GPU memory usage during training.

Limitations and Future Work

  • Visual Inspection for Semantics: While the channel-recurrent architecture shows promise for learning semantically meaningful latent spaces, determining the exact semantic meaning of specific latent directions still requires visual inspection.
  • Future Research Directions:
    • Building more interpretable features via channel recurrency.
  • Extrapolating the framework to other tasks beyond image modeling.
  • Exploring variations of the channel-recurrent architecture and its application in other generative models.
  • Investigating further methods for controlling factors of variation in the latent space.

Summary updated

BibTeX
@inproceedings{shang2018channel,
  author = {Wenling Shang and Kihyuk Sohn and Yuandong Tian},
  title = {{Channel-Recurrent Autoencoding for Image Modeling}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  url = {https://openreview.net/forum?id=g3MAoZqPibM},
}
Figure