Abstract
Model selection is crucial to high-dimensional learning and inference for contemporary big data applications in pinpointing the best set of covariates among a sequence of candidate interpretable models. Most existing work assumes implicitly that the models are correctly specified or have fixed dimensionality. Yet both features of model misspecification and high dimensionality are prevalent in practice. In this paper, we exploit the framework of model selection principles in misspecified models originated in Lv and Liu (2014) and investigate the asymptotic expansion of Bayesian principle of model selection in the setting of high-dimensional misspecified models. With a natural choice of prior probabilities that encourages interpretability and incorporates Kullback-Leibler divergence, we suggest the high-dimensional generalized Bayesian information criterion with prior probability (HGBIC$_p$) for large-scale model selection with misspecification. Our new information criterion characterizes the impacts of both model misspecification and high dimensionality on model selection. We further establish the consistency of covariance contrast matrix estimation and the model selection consistency of HGBIC$_p$ in ultra-high dimensions under some mild regularity conditions. The advantages of our new method are supported by numerical studies.
Introduction: The Challenge of High-Dimensional Realities
In the era of big data, researchers frequently encounter two simultaneous challenges: high dimensionality and model misspecification. High dimensionality ($p \gg n$) occurs when the number of potential predictors significantly exceeds the number of observations, a common scenario in genomics, finance, and social sciences. Model misspecification occurs because the simple parametric models we use—such as linear or generalized linear models (GLMs)—are often mere approximations of a much more complex, potentially non-linear, underlying reality.
Classical model selection tools like the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) were designed under the assumption that the "true" model is contained within the candidate set. When this assumption fails, or when the number of variables explodes, these traditional methods often falter, either by selecting overly complex models or by failing to identify the most relevant interpretable features. This paper introduces the High-dimensional Generalized Bayesian Information Criterion with prior probability (HGBIC$_p$) to bridge this gap, providing a robust framework for selection in the "wild west" of modern data.
Methodology: The HGBIC$_p$ Criterion
The authors derive HGBIC$_p$ by investigating the asymptotic expansion of Bayesian principles specifically within the context of high-dimensional, misspecified models. The resulting criterion is defined as:
$$\text{HGBIC}_p = -2\ell_n(\mathbf{y}, \hat{\boldsymbol{\beta}}_n) + 2(\log p^* )|\mathfrak{M}| + \text{tr}(\hat{\mathbf{H}}_n) - \log |\hat{\mathbf{H}}_n|$$
Key Components
- Log-Likelihood ($-2\ell_n$): This term measures the goodness-of-fit, rewarding models that better explain the observed data.
- The High-Dimensional Penalty ($2\log p^*$): Standard BIC uses a penalty of $\log n$. However, in high dimensions, this is insufficient to account for the massive search space. HGBIC$_p$ utilizes $p^* = pn^{1/2}$, imposing a much heavier penalty that accounts for the total number of available variables ($p$).
- The Covariance Contrast Matrix ($\mathbf{H}_n$): This represents the "secret sauce" for handling misspecification. The matrix $\mathbf{H}_n$ compares the variability of the model's score function under the assumed model against the true (but unknown) data distribution.
- If the model is correctly specified, $\mathbf{H}_n$ approaches the identity matrix, and the trace and log-determinant terms simplify.
- If the model is misspecified, these terms adjust the penalty, effectively measuring the "geometric distance" between the working model and the truth to ensure robust selection.
Theoretical Insights and Consistency
A primary contribution of this work is the proof of model selection consistency in ultra-high dimensions. The authors demonstrate that as the sample size $n$ grows, the probability that HGBIC$_p$ identifies the best possible sparse model approaches 100%, even if $p$ grows at an exponential rate relative to $n$.
This consistency is achieved through a strategic choice of prior probabilities. By using a prior that penalizes model size based on the total number of variables (e.g., $p^{-d}$), the criterion effectively suppresses the "false discovery" of irrelevant variables that often plagues high-dimensional inference. This approach encourages interpretability by favoring parsimonious models that capture the most significant effects.
Numerical Evidence
The performance of HGBIC$_p$ was evaluated using controlled simulations where the ground truth was known, allowing for a direct comparison with AIC, BIC, and the low-dimensional Generalized BIC (GBIC$_p$).
Multiple Index Model
In a scenario where the true relationship was non-linear but researchers fit a standard linear regression (a classic misspecification), HGBIC$_p$ showed remarkable resilience. As the number of variables ($p$) increased from 100 to 3,200, HGBIC$_p$ maintained a 100% success rate in identifying the correct variables. In the same high-dimensional setting, the standard BIC’s success rate plummeted to just 5%.
Logistic Regression with Interaction
When the true data-generating process included interaction terms but the working model only considered main effects, HGBIC$_p$ consistently produced the lowest prediction and classification errors. Furthermore, the criterion proved stable across various tuning parameters (controlled by a factor $\zeta$), suggesting it is highly practical for real-world applications where precise tuning is difficult.
Conclusion and Future Directions
HGBIC$_p$ provides a theoretically grounded and numerically robust solution for model selection when faced with the dual threats of high dimensionality and misspecification. While currently focused on Generalized Linear Models (GLMs), the framework offers a foundation for expansion into more complex structures, such as additive models or Cox proportional hazards models for survival analysis.
Future research may also explore the integration of HGBIC$_p$ with "knockoff" variables to further enhance controlled variable selection, potentially creating even more powerful tools for stable and reproducible inference in large-scale data analysis.
Summary updated