Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow

Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, Sergey Levine

In the world of adversarial learning, balance is everything. Whether you are training a Generative Adversarial Network (GAN) to create photorealistic faces or teaching a virtual humanoid to do a backflip via Imitation Learning, you are essentially managing a high-stakes game between a generator (the student) and a discriminator (the teacher).

The problem? The teacher is often too good. If the discriminator becomes too accurate too quickly, it provides "uninformative gradients"—essentially telling the student "you're wrong" without providing a clear path on how to improve. This leads to training instability and the dreaded "vanishing gradient" problem.

In their work, Peng et al. introduce the Variational Discriminator Bottleneck (VDB), a simple yet powerful regularization technique that "throttles" the discriminator's information flow to keep the learning process productive.

The Challenge of the Perfect Discriminator

Adversarial methods rely on the discriminator to distinguish between real data and generated samples. However, if the real and generated distributions are too different—which is typical at the start of training—a powerful discriminator can easily find a perfect separator.

Mathematically, this results in a decision boundary that is extremely sharp. For the generator, this is a nightmare: the gradient (the direction it needs to move to improve) becomes zero almost everywhere. Previous solutions like adding instance noise or applying gradient penalties help, but they are often difficult to tune. If the noise is too low, it does nothing; if it's too high, the discriminator becomes too weak to teach anything useful.

The Variational Discriminator Bottleneck (VDB)

To solve this, the authors propose the Variational Discriminator Bottleneck. Instead of the discriminator looking at the raw input $x$ directly, it evaluates a stochastic encoding $z$ produced by an encoder $E(z|x)$.

The core innovation is enforcing an Information Bottleneck:

  1. Compression: The method constrains the mutual information $I(X, Z)$ to be below a certain threshold $I_c$. This forces the discriminator to ignore irrelevant "noise" in the data and focus only on the most discerning features.
  2. Variational Bound: Since calculating mutual information directly is intractable, they use a variational approximation (similar to the KL-divergence term in a Variational Autoencoder) to penalize the encoding if it deviates too much from a simple prior, such as a Gaussian distribution.
  3. Adaptive Throttling: The system uses dual gradient descent to adaptively update the penalty weight ($\beta$). If the discriminator is absorbing too much information, $\beta$ increases to "throttle" it; if it's struggling, $\beta$ decreases.

This creates a strict "budget" for information. By setting this budget (e.g., $I_c < 1$ bit), the discriminator is physically prevented from reaching 100% accuracy, ensuring that gradients keep flowing to the generator.

Diverse Applications and Datasets

The authors evaluated VDB across three distinct domains, demonstrating its versatility:

VDB applied across diverse domains: motion imitation, image generation, and inverse RL.
  • Motion Imitation (VAIL): Mocap clips of a 34-DOF humanoid performing complex skills like backflips, cartwheels, dancing, running, and spinkicks.
  • Video Imitation: Learning to run using only raw RGB video of a 2D biped as a demonstration.
  • Inverse RL (VAIRL): 2D navigation tasks in "C-mazes" and "S-mazes" designed to test how well a learned reward function transfers to new environments.
  • Image Generation (VGAN): Standard benchmarks including CIFAR-10, CelebA ($128 \times 128$), and CelebAHQ ($1024 \times 1024$).

Key Insights

  • Smoothness via Compression: By limiting the "bits" the discriminator can use, the decision boundary is forced to be smoother. This ensures that even if a generated sample is far from the real data, the discriminator provides a gradient pointing toward the data manifold.
  • Robustness to High-Dimensional Noise: In video imitation, raw pixels contain massive amounts of irrelevant information (like backgrounds and lighting). The VDB naturally filters this out, allowing the agent to focus purely on the motion of the character.
  • Generalization in Inverse RL: Discriminators often overfit to the specific paths taken by an expert. The VDB forces the learned reward function to be parsimonious, making it much more effective when transferred to a mirrored or slightly different maze.

Key Results

Transferring learned reward functions in Inverse RL using VDB.
  • Transferable Rewards: In the S-maze transfer task, VAIRL achieved a mean return of 156.5, nearly matching the expert (153.2). In contrast, the previous state-of-the-art (AIRL) failed completely with a return of -0.14.
  • Motion Imitation: The Variational Adversarial Imitation Learning (VAIL) method significantly outperformed standard GAIL. For the "Spinkick" task, VAIL achieved an error of 0.34, compared to 1.07 for standard GAIL.
  • Learning from Video: VAIL was the only method capable of learning a running gait from raw video pixels. Standard GAIL and pixel-wise loss methods failed to produce any coherent motion.
  • GAN Stability: Combining VDB with gradient penalties (VGAN-GP) achieved an 18.1 FID on CIFAR-10, outperforming WGAN-GP (19.9) and standard GANs (63.6).

Limitations and Future Work

While the video imitation results are impressive, they were performed on synthetic scenes with clean backgrounds; testing on real-world, cluttered video remains a challenge. Additionally, while the method is more stable than instance noise, the information constraint $I_c$ is still a hyperparameter that needs to be set (though $0.5$ proved to be a robust default). Finally, while the authors provide analysis on non-degenerate gradients, a more rigorous theoretical proof of convergence under the VDB constraint is a promising direction for future work.

Summary updated

Abstract

Adversarial learning methods have been proposed for a wide range of applications, but the training of adversarial models can be notoriously unstable. Effectively balancing the performance of the generator and discriminator is critical, since a discriminator that achieves very high accuracy will produce relatively uninformative gradients. In this work, we propose a simple and general technique to constrain information flow in the discriminator by means of an information bottleneck. By enforcing a constraint on the mutual information between the observations and the discriminator's internal representation, we can effectively modulate the discriminator's accuracy and maintain useful and informative gradients. We demonstrate that our proposed variational discriminator bottleneck (VDB) leads to significant improvements across three distinct application areas for adversarial learning algorithms. Our primary evaluation studies the applicability of the VDB to imitation learning of dynamic continuous control skills, such as running. We show that our method can learn such skills directly from raw video demonstrations, substantially outperforming prior adversarial imitation learning methods. The VDB can also be combined with adversarial inverse reinforcement learning to learn parsimonious reward functions that can be transferred and re-optimized in new settings. Finally, we demonstrate that VDB can train GANs more effectively for image generation, improving upon a number of prior stabilization methods.
BibTeX
@inproceedings{peng2019variational,
  author = {Xue Bin Peng and Angjoo Kanazawa and Sam Toyer and Pieter Abbeel and Sergey Levine},
  title = {{Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=8-_8qm55Gnp},
}
Figure