Introduction
In many real-world regression tasks, the assumption of homoscedasticity—where noise is constant across the input space—is violated. Heteroscedasticity, where the noise level varies as a function of the input, requires models capable of estimating both the conditional mean and the conditional variance. While Heteroscedastic Kernel Ridge Regression (HKRR) provides a framework for this, standard maximum likelihood or least-squares approaches to variance estimation are fundamentally biased. Specifically, if the mean model overfits the training data, the resulting residuals appear smaller than the true underlying noise, leading to overly optimistic and narrow error bars. This paper introduces a method to correct this bias using leave-one-out (LOO) cross-validation.
Methodology
This paper proposes an approximately unbiased estimator for conditional variance by integrating LOO cross-validation into the variance-fitting process. The core algorithm utilizes an Iterative Re-weighted Least Squares (IRLS) procedure that alternates between two primary steps:
- Mean Estimation: Fitting a kernel model to the target values to determine the conditional mean $\mu(x)$.
- Variance Estimation: Fitting a second kernel model to the squared residuals to determine the conditional variance $\sigma^2(x)$.
The critical innovation lies in the second step. Instead of calculating residuals using the standard predicted mean $\hat{y}_i$, the authors use the LOO estimate $\hat{y}_{i}^{(-i)}$. This ensures that the variance model is trained on residuals that have not been artificially reduced by the mean model's tendency to overfit its training points.
To maintain computational efficiency, the authors avoid the naive $O(\ell^3)$ cost of retraining the model $\ell$ times. By leveraging the properties of the "hat matrix" $H$ and the Bartlett matrix inversion formula, all LOO estimates are computed in a single closed-form step with $O(\ell^2)$ complexity. This mathematical optimization makes the "honest" variance estimation nearly as fast as the biased standard approach, allowing it to be applied to medium-sized datasets without significant overhead.
Experimental Results
The method was evaluated using a synthetic benchmark originally proposed by Williams (1996). In this setup, the input $x$ is drawn from a uniform distribution on $(0, \pi)$, and the target $y$ is generated from a normal distribution where both the mean and the variance vary smoothly as sine functions. Because the ground truth is known, the authors can precisely measure the bias in the variance estimates.
As shown in Figure 4, the standard HKRR (panel c) consistently underestimates the conditional standard deviation, resulting in "optimistically narrow" credible intervals that fail to capture the true distribution of the data. In contrast, the LOOHKRR (panel d) produces a mean predicted standard deviation that is almost identical to the true value. Across 1,000 randomly generated datasets, the LOOHKRR estimates were shown to be approximately unbiased, providing a much more faithful representation of the model's actual uncertainty.
Conclusion
By substituting standard residuals with leave-one-out residuals, the proposed LOOHKRR method effectively mitigates the bias inherent in heteroscedastic variance estimation. This approach provides more reliable uncertainty quantification, which is crucial for safety-critical applications or scientific modeling where accurate error bars are as important as the predictions themselves. While the $O(\ell^2)$ complexity is efficient, future work may focus on scaling these techniques to high-dimensional "Big Data" scenarios and developing automated hyperparameter tuning for the variance kernel.
Summary updated