TL;DR
This paper introduces XAIC and FAIC, new model selection criteria that address a fundamental bias in AIC when test data inputs differ from training data inputs. By explicitly accounting for this input shift, XAIC and FAIC offer improved predictive performance, especially in scenarios like covariate shift or when predicting for specific new data points.
What Problem Does This Paper Address?
In supervised learning, model selection aims to find the best model for predicting future data. Traditional methods like AIC (Akaike Information Criterion) are designed to balance model complexity with goodness-of-fit. However, a critical assumption underlying AIC and many of its variants is that the test data inputs are drawn from the same distribution as, or are identical to, the training data inputs. This assumption often fails in real-world scenarios where the distribution of inputs can change (covariate shift) or when we're interested in predicting a specific new data point. When this assumption is violated, AIC can lead to selecting overly complex models, resulting in suboptimal predictions. This paper tackles this "input shift" problem, aiming to develop model selection criteria that are more robust to changes in input distributions and can better optimize for predictive performance in these realistic settings.
Methods
The core contribution of this paper is the development of two new model selection criteria: XAIC (Extra-sample AIC) and FAIC (Focused AIC). These methods are derived from a novel, unbiased expression for the extra-sample error, which explicitly accounts for the test input distribution.
- XAIC: This criterion extends AIC by incorporating a penalty term ($\kappa_{\chi'}$) that quantifies the dissimilarity between the test input distribution and the training input distribution. This term is derived from the Fisher information matrix and the covariance of the model's parameter estimates. XAIC aims to provide a better estimate of generalization error when test inputs may differ from training inputs, even if their distributions are the same.
- FAIC: This is a special case of XAIC designed for situations where we need to predict a specific new data point (i.e., a single test input $x'$). Instead of considering the entire test input distribution, FAIC uses the specific test input $x'$ at the model selection stage. This "focused" approach allows the criterion to select models that are particularly well-suited for predicting that specific $x'$, potentially leading to better performance for that particular prediction task.
- Small Sample Corrections: The authors also propose XAICC and FAICC, which are small-sample corrected versions of XAIC and FAIC, analogous to AICC.
The derivation relies on assumptions about the regularity of the models and estimators, similar to those used for AIC, but crucially relaxes the assumption of identical input distributions.
Datasets
The paper does not introduce new datasets. Instead, it uses simulated data for its experiments.
- Univariate Experiments: Linear models with unknown variance were considered, with polynomial degrees from 0 to 6. Training inputs were drawn from a standard Gaussian distribution (mean 0, variance 1). Output values were generated based on a true function (either $f_1(x) = x + 2$ or $f_2(x) = |x|$) plus Gaussian noise.
- Multivariate Experiments: Models corresponded to subsets of 6 input variables, with an intercept and variance parameter. The true function was a linear combination of these variables plus Gaussian noise. Training inputs were drawn from three different distributions: standard Gaussian, a uniform distribution, or a uniform "spike-and-slab" mixture of Gaussians. Test inputs were drawn from a standard Gaussian distribution. These multivariate experiments were designed to simulate mild covariate shift scenarios.
Key Insights and Ideas
- AIC's Input Bias: The paper convincingly demonstrates that AIC suffers from a bias when test inputs differ from training inputs, even if they come from the same distribution. This bias arises because AIC's penalty term doesn't account for the potential difficulty of predicting inputs that are "far" from the training data.
- XAIC's Input-Aware Penalty: The introduction of the $\kappa_{\chi'}$ term in XAIC is a key innovation. This term acts as an input-dependent penalty, effectively down-weighting models that might perform poorly on inputs dissimilar to those seen during training.
- FAIC's Predictive Focus: FAIC's ability to adapt model selection to a specific prediction point $x'$ is a powerful idea. It shifts the paradigm from selecting a single "best" model for all future data to selecting the best model for a particular future data point, which can lead to significant gains when prediction for specific instances is the goal.
- Trade-offs in Model Selection: The experiments highlight that while XAIC and FAIC offer improved predictive accuracy, especially in challenging scenarios, there can be trade-offs. FAIC, in particular, can sometimes have higher variance than AIC, and its focused nature might make interpretation harder.
Key Results
- Improved Predictive Performance: In experiments, XAIC and FAIC (and their small-sample corrected versions) consistently outperformed AIC, BIC, and Bayesian Model Averaging (BMA) in terms of squared risk, particularly when test inputs were significantly different from training inputs or when using the focused approach of FAIC.
- XAIC vs. AIC Bias: Proposition 3 mathematically shows that for random input data, AIC underestimates the model's extra-sample error on average, with the bias increasing with model complexity. XAIC, by accounting for input dissimilarity, corrects this bias.
- FAIC's Adaptability: FAIC demonstrated excellent risk performance across all test inputs in univariate experiments, adapting its model choice based on the proximity of the test input to the training data.
- Multivariate Success: In multivariate experiments, XAIC generally performed better than AIC. FAIC was particularly effective in scenarios with pronounced differences in training point density around different test points (e.g., spike-and-slab data).
Limitations and Future Work
- Interpretation vs. Prediction: The authors note that FAIC's focused approach, while beneficial for prediction, can make model interpretation more difficult compared to non-focused methods.
- Variance of FAIC: While FAIC can reduce bias, its estimate of generalization error might have a larger variance than AIC's, which can sometimes lead to worse performance.
- Generalizability: The paper focuses on linear models. Future work could explore extending XAIC and FAIC to more complex model classes, such as hierarchical Bayesian models.
- Other Model Selection Methods: The authors suggest investigating whether other model selection methods are affected by the same input-shift bias and if extra-sample or focused versions can be developed for them. BPIC is highlighted as a promising candidate for such an extension.
- Practical Usefulness: Further experiments are needed to confirm the practical usefulness of XAIC and FAIC across a wider range of real-world problems.
Summary updated