Introduction
Gaussian Processes (GPs) are powerful non-parametric Bayesian models. While mathematically elegant for regression tasks due to their Gaussian posteriors, applying them to classification introduces non-Gaussian likelihoods that complicate posterior calculations. This paper introduces a method to perform Bayesian classification with GPs using the Thouless-Anderson-Palmer (TAP) mean field method—a technique originating from the statistical mechanics of disordered systems. Crucially, this approach bypasses the need for prior knowledge of the input data distribution, offering a computationally efficient approximation for complex posteriors.
The Challenge: Classification with Gaussian Processes
In GP regression, the posterior is Gaussian, making inference straightforward. However, classification requires non-Gaussian likelihoods (such as step functions or sigmoids) to map continuous latent functions to discrete labels. This "squashing" effect necessitates solving high-dimensional integrals to compute the posterior distribution exactly.
Traditional approximations like Monte Carlo sampling are computationally heavy, while Laplace approximations struggle with non-smooth functions. The authors seek a robust, physics-grounded alternative that captures complex interactions between data points without the heavy computational burden of Markov Chain Monte Carlo (MCMC) methods.
Methodology: The TAP Approach
The authors adapt the TAP approach, originally designed to describe spin glasses in physics, to the realm of machine learning.
- Partition Function & Auxiliary Variables: The posterior distribution is expressed as a partition function ($Z$). To handle the inverse covariance matrix ($C^{-1}$), auxiliary variables are introduced, translating the problem into a familiar statistical mechanics framework.
- Beyond Naive Mean Field: Standard "Naive" Mean Field Theory assumes variables act independently under an average "field," which often fails in complex systems.
- The Onsager Reaction Term: The TAP method improves upon naive assumptions by adding an "Onsager reaction term" ($\Delta G$). This correction accounts for a variable's "reaction" on itself through its influence on other variables.
- Distribution Independence: Unlike previous TAP-based neural network models that required specific input data distributions (e.g., Gaussian inputs), this derivation utilizes a newer approach that makes the method applicable to arbitrary real-world datasets.
Key Insights
- Physics as a Proxy for Integration: The Bayesian inference problem is treated as a physical system reaching equilibrium. The system's "Free Energy" ($G$) corresponds to the negative log-likelihood of the evidence.
- Exactness in Regression: The TAP Mean Field Theory becomes exact when applied to standard Gaussian regression, indicating its high quality as an approximation for the GP framework.
- Efficient Leave-One-Out (LOO) Estimation: A standout feature of the TAP approach is its natural provision of an approximate LOO estimator for generalization error. Instead of retraining the model $N$ times, the LOO error can be calculated directly from the converged mean field equations.
- Robustness to Non-Smoothness: Unlike Laplace approximations, which rely on the likelihood being twice-differentiable, the TAP method effectively handles the non-smooth step functions used in binary classification.
Results on the Sonar Dataset
The method was evaluated on the Sonar (Mines vs. Rocks) dataset, where the task is to distinguish between sonar signals bounced off a metal cylinder versus a rock. The data was linearly rescaled to zero mean and unit variance to aid convergence.
As shown in the results table, the TAP GP (using a squared exponential kernel, denoted as covariance function 2) achieved a test error of 7.7%. This outperforms the best reported 2-layer neural network (9.6%) and significantly beats a simple perceptron (26.9%). Furthermore, the approximate LOO estimator provided by the TAP equations perfectly matched the exact LOO error (0.212 vs. 0.212), demonstrating the remarkable tightness of the approximation.
Limitations and Future Directions
While promising, the approach has a few limitations:
- Convergence Issues: The nonlinear equations require iterative solving. Occasionally, a diagonal "jitter" term (noise) must be added to the covariance matrix to ensure the algorithm converges.
- Hyperparameter Tuning: In this study, hyperparameters (like the kernel length scale) were manually tuned to minimize the LOO error. Future work could automate this by using the Free Energy ($G$) to optimize parameters via gradient descent.
- Optimization Landscape: Converting the current iterative equation-solving method into a direct minimization problem of the Free Energy could yield a more stable and efficient algorithm.
Summary updated