TL;DR
This paper introduces ARMS, a novel gradient estimator for binary variables that significantly reduces variance by using multiple, jointly antithetic samples. ARMS generalizes existing methods like REINFORCE, DisARM, and LOORF, and demonstrates superior performance in training generative models.
What Problem Does This Paper Address?
Estimating gradients for binary variables is crucial in many machine learning tasks, particularly for training discrete latent variable models. Traditional methods like REINFORCE often suffer from high variance, making training unstable and slow. While some methods use pairs of negatively correlated samples (like DisARM) or independent samples (like LOORF) to reduce variance, they don't fully leverage the potential of using more than two samples. The challenge is to develop an unbiased, low-variance gradient estimator that can effectively utilize an arbitrary number of samples for binary variables.
Methods
The core contribution is the ARMS (Antithetic REINFORCE Multi-Sample) gradient estimator. ARMS extends the idea of antithetic sampling (using negatively correlated samples) to n samples that are jointly antithetic. This is achieved by:
- Copula-based Sampling: ARMS utilizes copulas (specifically Dirichlet and Gaussian copulas) to generate n correlated Bernoulli samples that exhibit strong negative dependence. This ensures the samples are "antithetic" in a structured way.
- Generalization of Existing Methods: The paper shows that ARMS generalizes:
- DisARM/U2G: When n=2 and specific antithetic uniform samples are used.
- LOORF: When n independent samples are used (effectively, zero correlation).
- Multi-Sample Variational Bound Optimization: A variant of ARMS is developed for optimizing multi-sample variational bounds, outperforming VIMCO and DisARM.
The mathematical derivation involves extending the score function estimator with a multiplicative debiasing term that accounts for the pairwise correlation between the n samples.
Datasets
The authors evaluate ARMS on several standard datasets for training generative models:
- Dynamic MNIST
- Fashion MNIST
- Omniglot
These datasets are used with both linear and non-linear encoder-decoder architectures for Variational Autoencoders (VAEs).
Key Insights and Ideas
- The Power of Joint Antithetic Sampling: The key insight is that going beyond pairs of antithetic samples to n jointly antithetic samples, generated via copulas, leads to a significant and consistent reduction in gradient variance. This is more effective than simply using more independent samples or pairs of antithetic samples.
- ARMS as a Unifying Framework: ARMS elegantly unifies several existing gradient estimators (LOORF, DisARM) under a single framework, demonstrating their relationship and highlighting ARMS's broader applicability.
- Copula Choice Matters (Slightly): While both Dirichlet and Gaussian copulas are explored for generating correlated samples, the paper notes that for n=2, they produce the same antithetic pair. For larger n, the choice can influence the correlation structure, with the Dirichlet copula often showing slightly better performance in experiments.
- LOORF's Underappreciated Strength: The paper points out that LOORF, despite being a simpler baseline, performs surprisingly well, especially when the number of samples increases. This is because it effectively averages over many pairs, a benefit that ARMS aims to capture more efficiently.
Key Results
- Reduced Gradient Variance: Across various datasets and network architectures, ARMS consistently demonstrates significantly lower gradient variance compared to DisARM, LOORF, and RELAX. This is evident in the experimental plots (Figures 2, 3, 5, 6, 7, 8).
- Improved ELBO and Multi-Sample Bound Optimization: ARMS achieves better final ELBO scores and multi-sample bounds than competing methods on all tested datasets and network configurations (Tables 1 and 2). For instance, in Table 1, ARMS-D and ARMS-N often achieve the highest ELBO values.
- Competitive Performance with Fewer Samples: ARMS often outperforms other methods even when using fewer samples, highlighting its efficiency.
- Generalization: ARMS successfully generalizes LOORF and DisARM, showing its foundational nature.
Limitations and Future Work
- Sampling Correlated Bernoullis: While ARMS provides methods (Dirichlet and Gaussian copulas) for sampling correlated Bernoulli variables, the authors suggest that "better ways, not necessarily based on copulas" might exist.
- Extension to Categorical Variables: The paper mentions that ARMS is currently for binary variables. Extending it to categorical variables, similar to how ARM was extended to ARSM, is a promising future direction.
- Multi-Sample Bound Optimization: The paper optimized the multi-sample bound with i.i.d. samples for fair comparison. Exploring antithetic samples for this bound, which is also a lower bound of the marginal likelihood, is suggested.
Summary updated