ARMS: Antithetic-REINFORCE-Multi-Sample Gradient for Binary Variables

Alek Dimitriev, Mingyuan Zhou

TL;DR

This paper introduces ARMS, a novel gradient estimator for binary variables that significantly reduces variance by using multiple, jointly antithetic samples. ARMS generalizes existing methods like REINFORCE, DisARM, and LOORF, and demonstrates superior performance in training generative models.

What Problem Does This Paper Address?

Estimating gradients for binary variables is crucial in many machine learning tasks, particularly for training discrete latent variable models. Traditional methods like REINFORCE often suffer from high variance, making training unstable and slow. While some methods use pairs of negatively correlated samples (like DisARM) or independent samples (like LOORF) to reduce variance, they don't fully leverage the potential of using more than two samples. The challenge is to develop an unbiased, low-variance gradient estimator that can effectively utilize an arbitrary number of samples for binary variables.

Methods

The core contribution is the ARMS (Antithetic REINFORCE Multi-Sample) gradient estimator. ARMS extends the idea of antithetic sampling (using negatively correlated samples) to n samples that are jointly antithetic. This is achieved by:

  • Copula-based Sampling: ARMS utilizes copulas (specifically Dirichlet and Gaussian copulas) to generate n correlated Bernoulli samples that exhibit strong negative dependence. This ensures the samples are "antithetic" in a structured way.
  • Generalization of Existing Methods: The paper shows that ARMS generalizes:
  • DisARM/U2G: When n=2 and specific antithetic uniform samples are used.
  • LOORF: When n independent samples are used (effectively, zero correlation).
  • Multi-Sample Variational Bound Optimization: A variant of ARMS is developed for optimizing multi-sample variational bounds, outperforming VIMCO and DisARM.

The mathematical derivation involves extending the score function estimator with a multiplicative debiasing term that accounts for the pairwise correlation between the n samples.

Datasets

The authors evaluate ARMS on several standard datasets for training generative models:

  • Dynamic MNIST
  • Fashion MNIST
  • Omniglot

These datasets are used with both linear and non-linear encoder-decoder architectures for Variational Autoencoders (VAEs).

Key Insights and Ideas

  • The Power of Joint Antithetic Sampling: The key insight is that going beyond pairs of antithetic samples to n jointly antithetic samples, generated via copulas, leads to a significant and consistent reduction in gradient variance. This is more effective than simply using more independent samples or pairs of antithetic samples.
  • ARMS as a Unifying Framework: ARMS elegantly unifies several existing gradient estimators (LOORF, DisARM) under a single framework, demonstrating their relationship and highlighting ARMS's broader applicability.
  • Copula Choice Matters (Slightly): While both Dirichlet and Gaussian copulas are explored for generating correlated samples, the paper notes that for n=2, they produce the same antithetic pair. For larger n, the choice can influence the correlation structure, with the Dirichlet copula often showing slightly better performance in experiments.
  • LOORF's Underappreciated Strength: The paper points out that LOORF, despite being a simpler baseline, performs surprisingly well, especially when the number of samples increases. This is because it effectively averages over many pairs, a benefit that ARMS aims to capture more efficiently.

Key Results

  • Reduced Gradient Variance: Across various datasets and network architectures, ARMS consistently demonstrates significantly lower gradient variance compared to DisARM, LOORF, and RELAX. This is evident in the experimental plots (Figures 2, 3, 5, 6, 7, 8).
  • Improved ELBO and Multi-Sample Bound Optimization: ARMS achieves better final ELBO scores and multi-sample bounds than competing methods on all tested datasets and network configurations (Tables 1 and 2). For instance, in Table 1, ARMS-D and ARMS-N often achieve the highest ELBO values.
  • Competitive Performance with Fewer Samples: ARMS often outperforms other methods even when using fewer samples, highlighting its efficiency.
  • Generalization: ARMS successfully generalizes LOORF and DisARM, showing its foundational nature.

Limitations and Future Work

  • Sampling Correlated Bernoullis: While ARMS provides methods (Dirichlet and Gaussian copulas) for sampling correlated Bernoulli variables, the authors suggest that "better ways, not necessarily based on copulas" might exist.
  • Extension to Categorical Variables: The paper mentions that ARMS is currently for binary variables. Extending it to categorical variables, similar to how ARM was extended to ARSM, is a promising future direction.
  • Multi-Sample Bound Optimization: The paper optimized the multi-sample bound with i.i.d. samples for fair comparison. Exploring antithetic samples for this bound, which is also a lower bound of the marginal likelihood, is suggested.

Summary updated

Abstract

Estimating the gradients for binary variables is a task that arises frequently in various domains, such as training discrete latent variable models. What has been commonly used is a REINFORCE based Monte Carlo estimation method that uses either independent samples or pairs of negatively correlated samples. To better utilize more than two samples, we propose ARMS, an Antithetic REINFORCE-based Multi-Sample gradient estimator. ARMS uses a copula to generate any number of mutually antithetic samples. It is unbiased, has low variance, and generalizes both DisARM, which we show to be ARMS with two samples, and the leave-one-out REINFORCE (LOORF) estimator, which is ARMS with uncorrelated samples. We evaluate ARMS on several datasets for training generative models, and our experimental results show that it outperforms competing methods. We also develop a version of ARMS for optimizing the multi-sample variational bound, and show that it outperforms both VIMCO and DisARM. The code is publicly available.
BibTeX
@inproceedings{dimitriev2021arms,
  author = {Alek Dimitriev and Mingyuan Zhou},
  title = {{ARMS: Antithetic-REINFORCE-Multi-Sample Gradient for Binary Variables}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=GQJz5-Dd4d},
}
Figure