TL;DR
The authors introduce the Amortized Residual Denoising Autoencoder (AR-DAE), a novel method for estimating entropy gradients in continuous random variables. By combining a residual architecture with noise-level amortization, AR-DAE overcomes the numerical instability and optimization hurdles of standard denoising autoencoders, enabling state-of-the-art performance in variational inference and reinforcement learning.
What Problem Does This Paper Address?
Entropy is a fundamental concept in machine learning, used to encourage exploration in Reinforcement Learning (RL), prevent mode collapse in Generative Adversarial Networks (GANs), and calibrate uncertainty in Bayesian inference. However, for arbitrary continuous distributions—especially "implicit" ones defined by a generator function—computing entropy is generally intractable.
Researchers often rely on the Denoising Autoencoder (DAE) to estimate the gradient of the log-density (the "score"), which can then be used to estimate the entropy gradient. However, DAEs face a "no-win" trade-off:
- Accuracy requires low noise: The DAE's reconstruction function only converges to the true score as the injected noise level ($\sigma$) approaches zero.
- Training requires high noise: As $\sigma \to 0$, the training signal vanishes, leading to high optimization error and numerical instability (due to dividing by $\sigma^2$).
This paper aims to break this trade-off to provide a stable, low-variance, and theoretically unbiased estimator for entropy gradients.
Methods
The proposed AR-DAE incorporates three key architectural and training innovations:
- Residual Form: Instead of a standard autoencoder $r(x)$, they parameterize the model as $r(x) = x + \sigma^2 f_{ar}(x)$. This allows the network $f_{ar}$ to directly output the log-density gradient, eliminating the numerically unstable division by small $\sigma^2$ during inference.
- Amortization (Scale Conditioning): The model $f_{ar}(x; \sigma)$ is conditioned on the noise level $\sigma$. During training, $\sigma$ is sampled from a distribution (e.g., a centered Gaussian). This allows the network to learn from a wide range of noise levels simultaneously. At inference time, they simply set $\sigma = 0$ to recover the "clean" log-density gradient.
- Loss Rescaling: The standard DAE loss is rescaled by $1/\sigma$ to prevent the gradient signal from vanishing at small noise levels, ensuring stable optimization even as the model approaches the zero-noise limit.
Datasets
The authors evaluate AR-DAE across several domains:
- Toy Distributions: 1D Mixture of Gaussians (MoG), 2D Swissroll, and a 2D 5x5 grid of Gaussians to analyze approximation errors and visualize gradient fields.
- Image Benchmarks: Dynamically and statically binarized MNIST for Variational Autoencoder (VAE) experiments.
- Reinforcement Learning: Continuous control environments from OpenAI Gym (HalfCheetah, Ant, Hopper, Walker, Humanoid) and Rllab (Humanoid).
Key Insights and Ideas
- Interpolation vs. Extrapolation: By training on a range of $\sigma$ values, the network effectively interpolates between a simple Gaussian gradient field (at high $\sigma$) and the complex data manifold (at low $\sigma$). This "curriculum" makes learning difficult distributions more robust.
- Breaking the Vanishing Signal: The combination of residual parameterization and loss rescaling ensures that even when the noise is nearly zero, the network receives a meaningful training signal to refine its estimate of the score.
- Robustness to Non-Stationarity: In tasks like RL or VAE training, the data distribution is constantly changing. AR-DAE's amortized nature allows it to adapt more quickly and accurately than standard DAEs that might be tuned for a specific, fixed noise level.
Key Results
- Lower Approximation Error: Ablation studies on a 1D MoG show that AR-DAE achieves significantly lower expected error in score estimation compared to regular DAEs and annealed versions, closely following the theoretical lower bound.
- State-of-the-Art VAEs: On statically binarized MNIST, VAEs trained with AR-DAE achieved a log-likelihood of -79.61, outperforming other implicit posterior methods like AVB and NAF, and rivaling models with complex trained priors.
- Improved Continuous Control: Integrating AR-DAE into the Soft Actor-Critic (SAC) algorithm led to superior performance on high-dimensional tasks. For example, on the Humanoid (rllab) environment, SAC-AR-DAE achieved a maximum average return of 10739, nearly doubling the performance of standard SAC (6099) and significantly outperforming Normalizing Flow-based versions.
Limitations and Future Work
- Computational Overhead: AR-DAE requires multiple updates per generator update to stay "up-to-date" with the evolving data distribution, which increases training time.
- Heuristic Sensitivity: The authors note that smooth activation functions (like Softplus or ELU) are crucial; standard ReLUs performed poorly in low-density regions.
- Future Directions: The authors suggest exploring more complex noise distributions for amortization and applying AR-DAE to other implicit models where entropy gradients are currently a bottleneck.
Summary updated