AR-DAE: Towards Unbiased Neural Entropy Gradient Estimation

Jae Hyun Lim, Aaron Courville, Christopher Pal, Chin-Wei Huang

TL;DR

The authors introduce the Amortized Residual Denoising Autoencoder (AR-DAE), a novel method for estimating entropy gradients in continuous random variables. By combining a residual architecture with noise-level amortization, AR-DAE overcomes the numerical instability and optimization hurdles of standard denoising autoencoders, enabling state-of-the-art performance in variational inference and reinforcement learning.

What Problem Does This Paper Address?

Entropy is a fundamental concept in machine learning, used to encourage exploration in Reinforcement Learning (RL), prevent mode collapse in Generative Adversarial Networks (GANs), and calibrate uncertainty in Bayesian inference. However, for arbitrary continuous distributions—especially "implicit" ones defined by a generator function—computing entropy is generally intractable.

Researchers often rely on the Denoising Autoencoder (DAE) to estimate the gradient of the log-density (the "score"), which can then be used to estimate the entropy gradient. However, DAEs face a "no-win" trade-off:

  1. Accuracy requires low noise: The DAE's reconstruction function only converges to the true score as the injected noise level ($\sigma$) approaches zero.
  2. Training requires high noise: As $\sigma \to 0$, the training signal vanishes, leading to high optimization error and numerical instability (due to dividing by $\sigma^2$).

This paper aims to break this trade-off to provide a stable, low-variance, and theoretically unbiased estimator for entropy gradients.

Methods

The proposed AR-DAE incorporates three key architectural and training innovations:

  • Residual Form: Instead of a standard autoencoder $r(x)$, they parameterize the model as $r(x) = x + \sigma^2 f_{ar}(x)$. This allows the network $f_{ar}$ to directly output the log-density gradient, eliminating the numerically unstable division by small $\sigma^2$ during inference.
  • Amortization (Scale Conditioning): The model $f_{ar}(x; \sigma)$ is conditioned on the noise level $\sigma$. During training, $\sigma$ is sampled from a distribution (e.g., a centered Gaussian). This allows the network to learn from a wide range of noise levels simultaneously. At inference time, they simply set $\sigma = 0$ to recover the "clean" log-density gradient.
  • Loss Rescaling: The standard DAE loss is rescaled by $1/\sigma$ to prevent the gradient signal from vanishing at small noise levels, ensuring stable optimization even as the model approaches the zero-noise limit.
Figure 2. Residual DAE trained with a large (left) vs small (right) σ value. Red cross indicates the mean of the swissroll. The arrows indicate the approximate gradient directions.

Datasets

The authors evaluate AR-DAE across several domains:

  • Toy Distributions: 1D Mixture of Gaussians (MoG), 2D Swissroll, and a 2D 5x5 grid of Gaussians to analyze approximation errors and visualize gradient fields.
  • Image Benchmarks: Dynamically and statically binarized MNIST for Variational Autoencoder (VAE) experiments.
  • Reinforcement Learning: Continuous control environments from OpenAI Gym (HalfCheetah, Ant, Hopper, Walker, Humanoid) and Rllab (Humanoid).

Key Insights and Ideas

  • Interpolation vs. Extrapolation: By training on a range of $\sigma$ values, the network effectively interpolates between a simple Gaussian gradient field (at high $\sigma$) and the complex data manifold (at low $\sigma$). This "curriculum" makes learning difficult distributions more robust.
  • Breaking the Vanishing Signal: The combination of residual parameterization and loss rescaling ensures that even when the noise is nearly zero, the network receives a meaningful training signal to refine its estimate of the score.
  • Robustness to Non-Stationarity: In tasks like RL or VAE training, the data distribution is constantly changing. AR-DAE's amortized nature allows it to adapt more quickly and accurately than standard DAEs that might be tuned for a specific, fixed noise level.
Figure 3. Approximation error of different DAE variants. AR-DAE (blue) significantly reduces error as σ approaches zero compared to baselines.

Key Results

  • Lower Approximation Error: Ablation studies on a 1D MoG show that AR-DAE achieves significantly lower expected error in score estimation compared to regular DAEs and annealed versions, closely following the theoretical lower bound.
  • State-of-the-Art VAEs: On statically binarized MNIST, VAEs trained with AR-DAE achieved a log-likelihood of -79.61, outperforming other implicit posterior methods like AVB and NAF, and rivaling models with complex trained priors.
  • Improved Continuous Control: Integrating AR-DAE into the Soft Actor-Critic (SAC) algorithm led to superior performance on high-dimensional tasks. For example, on the Humanoid (rllab) environment, SAC-AR-DAE achieved a maximum average return of 10739, nearly doubling the performance of standard SAC (6099) and significantly outperforming Normalizing Flow-based versions.
Figure 4. Fitting energy functions. First column: target energy functions. Second column: auxiliary variational method for hier- archical model. Third column: hierarchical model trained with AR-DAE. Last column: implicit model trained with AR-DAE.

Limitations and Future Work

  • Computational Overhead: AR-DAE requires multiple updates per generator update to stay "up-to-date" with the evolving data distribution, which increases training time.
  • Heuristic Sensitivity: The authors note that smooth activation functions (like Softplus or ELU) are crucial; standard ReLUs performed poorly in low-density regions.
  • Future Directions: The authors suggest exploring more complex noise distributions for amortization and applying AR-DAE to other implicit models where entropy gradients are currently a bottleneck.

Summary updated

Abstract

Entropy is ubiquitous in machine learning, but it is in general intractable to compute the entropy of the distribution of an arbitrary continuous random variable. In this paper, we propose the amortized residual denoising autoencoder (AR-DAE) to approximate the gradient of the log density function, which can be used to estimate the gradient of entropy. Amortization allows us to significantly reduce the error of the gradient approximator by approaching asymptotic optimality of a regular DAE, in which case the estimation is in theory unbiased. We conduct theoretical and experimental analyses on the approximation error of the proposed method, as well as extensive studies on heuristics to ensure its robustness. Finally, using the proposed gradient approximator to estimate the gradient of entropy, we demonstrate state-of-the-art performance on density estimation with variational autoencoders and continuous control with soft actor-critic.
BibTeX
@inproceedings{lim2020ar,
  author = {Jae Hyun Lim and Aaron Courville and Christopher Pal and Chin-Wei Huang},
  title = {{AR-DAE: Towards Unbiased Neural Entropy Gradient Estimation}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
}
Figure