Abstract
This paper presents a practical framework for Gaussian Process Classifier (GPC) model selection using Leave-One-Out Cross-Validation (LOO-CV). By leveraging Expectation Propagation (EP) to approximate intractable predictive distributions and "smoothing" discrete metrics like the F-measure, the authors provide a gradient-based optimization method that significantly outperforms standard approaches on imbalanced datasets.
Introduction
In Gaussian Process (GP) modeling, model selection typically involves choosing optimal hyperparameters, such as kernel width or signal variance. The conventional approach is maximizing the Marginal Likelihood (Evidence Maximization). While mathematically elegant, Marginal Likelihood does not always align with specific real-world goals, particularly in the context of imbalanced data where one class is significantly rarer than the other. In such scenarios, practitioners often prioritize metrics like the F-measure or Weighted Error Rate. However, these metrics are discrete and non-differentiable, making them unsuitable for standard gradient-based optimization tools. Furthermore, calculating Leave-One-Out (LOO) error for GP classifiers is traditionally computationally expensive. This research addresses these challenges by providing a differentiable framework for LOO-CV optimization.
Methodology: The EP-CV Algorithm
The proposed EP-CV algorithm combines several techniques to enable efficient model selection based on predictive performance:
- EP Approximation: Expectation Propagation (EP) is utilized to approximate the posterior distribution, offering higher accuracy for classification tasks compared to the Laplace approximation.
- Tractable LOO via Cavity Distributions: The algorithm utilizes "cavity distributions" generated during the EP process. These distributions provide a computationally efficient way to approximate LOO predictive distributions without the need to retrain the model $n$ times.
- Metric Smoothing: To optimize for non-differentiable metrics like the F-measure, the authors replace discrete indicator functions with sigmoid-like probabilistic functions. This transformation creates a "smoothed" objective function with useful gradients for optimization.
- EM-like Optimization: The framework employs an Expectation-Maximization (EM) style approach. In the "E-step," the best EP site parameters are determined. In the "M-step," these parameters are held fixed while hyperparameters are optimized using the gradients of the smoothed LOO criteria.
Experimental Evaluation
The authors evaluated their approach across two distinct benchmark sets:
- Standard Benchmarks: 13 UCI-style datasets (e.g., Breastcancer, Diabetes, German, Titanic) were used to test general performance.
- Imbalanced Data Benchmarks: 8 datasets (e.g., Yeast variants, Car3, Ecoli5) were specifically selected or modified to evaluate performance on imbalanced classification tasks.
Key Insights
A central argument of the paper is that for specific tasks—such as medical diagnostics where false negatives carry high costs—models should be optimized directly for the target evaluation metric rather than just maximizing training data probability. The use of EP "cavities" allows for LOO-CV optimization at a computational cost comparable to standard Marginal Likelihood. By smoothing the F-measure, the authors transform a jagged, non-differentiable landscape into a navigable surface for gradient-based optimizers.
Results and Discussion
The EP-CV approach demonstrated competitive performance on standard tasks, matching or slightly exceeding Marginal Likelihood (EP-ML) and significantly outperforming Least Squares-based LOO methods. The primary advantage was observed in imbalanced datasets. For instance, on the Yeast7 dataset (where the positive class represents only 2% of the data), the EP-CV(FM) method showed substantial improvements in the F-measure compared to models optimized via standard likelihood. The smoothed F-measure was also shown to increase monotonically during optimization, indicating a stable and reliable training process.
Limitations and Future Work
Despite its advantages, the algorithm's $O(n^3)$ scaling limits its application to very large datasets without further sparse approximations. Additionally, the current optimization assumes fixed site parameters during hyperparameter updates; accounting for implicit derivatives might improve accuracy but adds significant complexity. Future research could also focus on the automatic tuning of the bias parameter $\gamma$ in conjunction with kernel parameters.
Summary updated