Abstract: We propose a conjugate and calibrated Gaussian process (GP) model for multi-class classification by exploiting the geometry of the probability simplex. Our approach uses Aitchison geometry to map simplex-valued class probabilities to an unconstrained Euclidean representation, turning classification into a GP regression problem with fewer latent dimensions than standard multi-class GP classifiers. This yields conjugate inference and reliable predictive probabilities without relying on distributional approximations in the model construction. The method is compatible with standard sparse GP regression techniques, enabling scalable inference on larger datasets. Empirical results show well-calibrated and competitive performance across synthetic and real-world datasets.
Introduction
Gaussian Processes (GPs) are widely regarded as a premier framework for uncertainty quantification in machine learning. However, their application to multiclass classification is hindered by a fundamental lack of conjugacy between Gaussian priors and standard classification likelihoods, such as the softmax function. This non-conjugacy necessitates the use of approximations—like Variational Inference or MCMC—which are often computationally intensive and can lead to poor calibration, resulting in models that are overconfident or underconfident.
Furthermore, standard GP classifiers typically require $K$ latent functions for $K$ classes. This is inherently redundant because class probabilities must sum to one, meaning they reside on a $(K-1)$-dimensional simplex. This paper introduces Exact-ILR, a geometric approach that transforms multiclass classification into a straightforward GP regression problem. By leveraging Aitchison geometry, specifically the Isometric Log-Ratio (ILR) transform, the authors map the probability simplex into an unconstrained Euclidean space, allowing for exact (conjugate) inference and superior uncertainty calibration.
The Exact-ILR Framework
The core of the proposed method lies in bridging the gap between the probability simplex $\Delta^D$ (where $D=K-1$) and Euclidean space $\mathbb{R}^D$.
The Exact-ILR framework follows a three-step process:
- Label Smoothing: Discrete class labels are "smoothed" into the interior of the simplex using a hyperparameter $\lambda$.
- ILR Mapping: These smoothed labels are transformed via the ILR bijection into "pseudo-observations" in $\mathbb{R}^D$.
- Conjugate Regression: Standard GP regression is performed on these pseudo-observations. Since the likelihood in this transformed space is Gaussian, the posterior inference is exact and conjugate.
To make predictions, the Gaussian predictive distribution from the GP is mapped back to the simplex using the inverse ILR transform. For large datasets, the method integrates seamlessly with sparse GP techniques (e.g., using inducing points), maintaining a computational complexity of $O(NM^2)$.
Experimental Evaluation
The authors evaluated Exact-ILR against several baselines, including Dirichlet-based GPs (GPD) and Variational GP classifiers (SVGPC), across a range of UCI benchmarks and synthetic datasets.
Key findings from the empirical study include:
- Superior Calibration: Exact-ILR consistently achieved the lowest Expected Calibration Error (ECE) across almost all datasets. This ensures that the model's confidence levels closely match its actual accuracy.
- Competitive Accuracy: The method matches or exceeds the accuracy of state-of-the-art models while providing better Negative Log-Likelihood (NLL) scores, particularly on complex datasets with many classes, such as the 26-class LETTER dataset.
- Dimensional Efficiency: By operating in $K-1$ dimensions, the model is more parsimonious than standard $K$-dimensional GP classifiers.
- Robustness and Scalability: Unlike Dirichlet-based methods that can degrade as the number of classes increases due to approximation noise, Exact-ILR remains stable and accurate even when scaled to hundreds of classes.
Discussion and Future Work
While Exact-ILR offers significant advantages in calibration and conjugacy, it relies on a label-smoothing parameter $\lambda$. While effective, future work could explore automated methods for tuning this parameter or developing input-dependent noise models to better handle regions of high class overlap. Additionally, exploring multi-output kernels to capture correlations between classes directly in the ILR space remains an open area for research.
By re-framing classification through the lens of Aitchison geometry, Exact-ILR provides a principled, efficient, and well-calibrated solution for multiclass Gaussian Process modeling.
Summary updated