Abstract
Accurate outlier detection is not only a necessary preprocessing step, but can itself give important insights into the data. However, especially for non-linear regression, the detection of outliers is non-trivial and actually ambiguous. We propose a new method that identifies outliers by finding a subset of data points $T$ such that the marginal likelihood of all remaining data points $S$ is maximized. Though the idea is more general, it is particularly appealing for Gaussian process regression, where the marginal likelihood has an analytic solution. While maximizing the marginal likelihood for hyper-parameter optimization is a well-established non-convex optimization problem, optimizing the set of data points $S$ is not. Indeed, even a greedy approximation is computationally challenging due to the high cost of evaluating the marginal likelihood. As a remedy, we propose an efficient projected gradient descent method with provable convergence guarantees. Moreover, we also establish the breakdown point when jointly optimizing hyper-parameters and $S$. For various datasets and types of outliers, our experiments demonstrate that the proposed method can improve outlier detection and robustness when compared with several popular alternatives like the student-t likelihood.
Introduction
Gaussian Processes (GPs) are powerful tools for non-linear regression, but they are notoriously sensitive to outliers. Because standard GPs assume Gaussian noise, a single extreme data point can pull the entire regression curve away from the true underlying pattern. Existing solutions, such as using a student-t likelihood, often sacrifice the analytic tractability of the marginal likelihood or require manual tuning of complex hyperparameters. Furthermore, identifying outliers in non-linear spaces is inherently ambiguous. This paper seeks a principled, automated way to "trim" away bad data to reveal the true signal.
Methods: The $\nu$-Trimmed Marginal Likelihood
We propose the $\nu$-trimmed marginal likelihood approach. Instead of fitting the model to all $n$ data points, the algorithm seeks a subset $S$ of size $m$ (where $m < n$) that maximizes the marginal likelihood. The remaining points are treated as outliers.
Key technical contributions include:
- Joint Optimization: The method simultaneously optimizes GP hyperparameters (e.g., length-scales) and the selection of the "inlier" set $S$.
- Projected Gradient Descent (PGD): Since choosing the best subset $S$ is an NP-hard combinatorial problem, we reformulate it using auxiliary correction variables and use PGD to find a solution. We provide theoretical proofs that this approach converges to a stationary point.
- Breakdown Point Analysis: We prove that the trimming factor $\nu$ corresponds to the "breakdown point"—the minimum ratio of contamination needed to break the model.
- Iterative $\nu$ Estimation: If the outlier ratio $\nu$ is unknown, we provide an iterative algorithm using $k$-fold cross-validation to estimate it automatically.
Results and Discussion
The proposed $\nu$-GP was evaluated against standard GPs, student-t GPs ($t$-GP), and $\gamma$-divergence methods ($\gamma$-GP) across synthetic and real-world benchmarks.
Handling Focused Outliers
One of the most striking insights is how the method handles "focused" outliers—clumps of bad data. Standard robust methods like $t$-GPs often get "tricked" into interpreting a clump of outliers as a real local feature of the function. The trimmed likelihood approach is significantly better at identifying these as a separate, ignorable group.
On the focused outlier task, the $\nu$-GP achieved an R-precision score of 0.97, compared to just 0.20 for the $t$-GP.
Performance on Real-World Data
In tests across various datasets from the LIBSVM archive (e.g., bodyfat, housing, spacega), the $\nu$-GP consistently outperformed or matched state-of-the-art robust regression techniques in terms of both outlier detection (R-precision) and predictive accuracy (RMSE).
Computational Efficiency
The PGD optimization proved to be orders of magnitude faster than greedy search methods. For the spacega dataset, PGD took 0.71 minutes, while a standard greedy approach took over 12 minutes.
Conclusion
The $\nu$-trimmed marginal likelihood offers a robust and computationally efficient framework for Gaussian Process regression. By leveraging the analytic properties of the GP marginal likelihood and an efficient PGD solver, the method provides superior outlier detection and predictive performance, particularly in challenging scenarios like focused outliers. Future work will address the inherent ambiguity in defining outliers in non-parametric regression and extend the framework to handle outliers in covariates.
Summary updated