Abstract
The logistic regression model is the most popular model for analyzing binary data. In the absence of any prior information, an improper flat prior is often used for the regression coefficients in Bayesian logistic regression models. The resulting intractable posterior density can be explored by running Polson et al.'s (2013) data augmentation (DA) algorithm. In this paper, we establish that the Markov chain underlying Polson et al.'s (2013) DA algorithm is geometrically ergodic. Proving this theoretical result is practically important as it ensures the existence of central limit theorems (CLTs) for sample averages under a finite second moment condition. The CLT in turn allows users of the DA algorithm to calculate standard errors for posterior estimates.
Introduction
In Bayesian statistics, estimating the coefficients ($\beta$) of a logistic regression model often requires handling intractable posterior distributions. When researchers lack specific prior knowledge, they frequently turn to a "flat prior," which assigns equal weight to all possible coefficient values. To sample from the resulting posterior, the Polya-Gamma (PG) data augmentation algorithm, introduced by Polson et al. (2013), has become the industry standard due to its efficiency and ease of implementation.
However, a critical theoretical gap has persisted: while the PG sampler was known to eventually converge to the correct distribution, the rate of that convergence remained unproven for the flat prior case. Without establishing "geometric ergodicity"—a property where the Markov chain converges to its stationary distribution at a geometric rate—practitioners could not be mathematically certain that the Central Limit Theorem (CLT) applied to their simulations. This paper fills that gap, providing the rigorous proof necessary to justify the calculation of standard errors in these models.
Methodology
The proof centers on the construction of a Markov chain underlying the PG Gibbs sampler. The authors utilize a classic tool from Markov chain theory known as a drift condition.
The Drift Function
The researchers constructed a specialized drift function, $V(\omega)$, designed to measure the distance of the algorithm's latent variables from the center of the distribution. By employing sophisticated linear algebra and matrix inequalities, the authors demonstrated that at each step of the Gibbs sampler, the expectation of this function "drifts" back toward a central set at a geometric rate.
Matrix Inequalities and Design Matrices
A significant portion of the proof involves handling the relationship between the design matrix $X$ (the observed data) and the latent variables introduced by the Polya-Gamma transformation. Unlike previous studies that focused on specific subsets like ANOVA models or informative Normal priors, this work addresses the general case of logistic regression. The proof holds for any dataset $(X, y)$ provided the design matrix $X$ has full rank and the resulting posterior distribution is mathematically "proper."
Key Insights and Theoretical Results
Propriety and Performance
One of the most significant findings is the alignment between model validity and algorithmic performance. The conditions required for the PG sampler to converge quickly are identical to the conditions required for the Bayesian posterior to be proper. Essentially, if the model is mathematically valid, the algorithm is guaranteed to be efficient.
Solidarity of the Chain
The authors established a "solidarity" property between the latent variables ($\omega$) and the regression coefficients ($\beta$). They proved that if the latent variable chain converges geometrically, the chain for the coefficients must also converge at the same rate. This ensures that the entire Gibbs sampling system is robust.
Theorem 1: Formal Proof of Geometric Ergodicity
The primary contribution of the paper is Theorem 1, which formally establishes that the PG Gibbs sampler for Bayesian logistic regression with a flat prior is geometrically ergodic. This result provides the theoretical foundation for using "batch means" or "spectral variance" methods to calculate the uncertainty of posterior estimates.
Practical Implications
For practitioners, this result is a significant "win." It moves the PG sampler from a heuristically successful tool to a theoretically sound one. By guaranteeing the existence of a CLT, researchers can now confidently report standard errors for their model estimates, ensuring that the precision of scientific results in fields ranging from medicine to social sciences is statistically grounded.
Limitations and Future Directions
While this paper provides a definitive answer for flat priors, it also highlights areas for further study:
- Alternative Priors: The proof is specifically tailored to improper flat priors. Other "objective" priors may require distinct mathematical treatments to establish similar convergence rates.
- Mixed Models: The current work focuses on fixed-effects logistic regression. Extending these proofs to Bayesian logistic mixed models (which include random effects) remains a complex challenge, as the convergence properties under improper priors in those settings are not yet fully understood.
- Technique Specificity: The drift function used is highly specialized to the Polya-Gamma distribution, suggesting that other data augmentation schemes will require unique approaches to prove geometric ergodicity.
Summary updated