Model Selection in High-Dimensional Misspecified Models

Pallavi Basu, Yang Feng, Jinchi LvarXiv 2014 · DOI · arXiv

TL;DR

This paper tackles the "double whammy" of modern statistics: selecting the right variables when you have way more features than samples ($p \gg n$) and your candidate models are fundamentally "wrong" (misspecified). The authors introduce $GBIC_p$, a new model selection criterion that explicitly adjusts for model bias and the massive search space of high-dimensional data, significantly outperforming classical tools like AIC and BIC.

What Problem Does This Paper Address?

In the "Big Data" era, we often deal with datasets where the number of variables (genes, sensors, economic indicators) far exceeds the number of observations. Standard practice is to assume sparsity—that only a few variables actually matter—and use criteria like AIC (Akaike Information Criterion) or BIC (Bayesian Information Criterion) to find them.

However, these classical tools rely on two dangerous assumptions:

  1. Correct Specification: They assume the "true" model (e.g., a perfect linear relationship) is among the candidates you are testing.
  2. Fixed Dimensions: They were designed for cases where the number of variables is small and stays constant as you collect more data.

In reality, models are almost always misspecified (e.g., we fit a linear model to a non-linear process), and the number of variables we track is exploding. When these two conditions collide, traditional AIC and BIC become inconsistent, often selecting models that are far too complex or missing the signal entirely.

Methods

The authors bridge the gap between misspecification theory (traditionally low-dimensional) and high-dimensional statistics.

Key components of their approach include:

  • Quasi-Maximum Likelihood Estimation (QMLE): Since the authors acknowledge the model is likely wrong, they use QMLE to find the "best possible" parameters within a flawed model family.
  • The Covariance Contrast Matrix ($H_n$): They define $H_n = A_n^{-1} B_n$, where $A_n$ is the Hessian of the log-likelihood (representing the model's internal geometry) and $B_n$ is the variance of the score function (representing the true data-generating process). If the model were perfect, $H_n$ would be the identity matrix. The "trace" and "determinant" of this matrix serve as a correction factor for the model's "wrongness."
  • Generalized BIC with Prior Probability ($GBIC_p$): The headline contribution. They derive a Bayesian expansion that includes a penalty term involving $\log p$ (the total number of features). This heavier penalty accounts for the sheer difficulty of finding the right needle in a massive haystack of $p$ variables.

Datasets

The paper validates the new criteria through both rigorous simulations and real-world biological data:

  • Simulated Scenarios:
    • Linear regression with hidden interaction terms (making the fitted linear model misspecified).
  • Multiple index models with non-linear functions.
  • Logistic regression with interaction terms.
  • Dimensionality was scaled from $p=200$ up to $p=3200$, with a constant sample size of $n=100$ or $n=200$.
  • Real-World Data:
    • Prostate Cancer Dataset: 12,601 genes across 136 samples.
  • Neuroblastoma (NB) Dataset: 10,707 genes across 239 patients.

Key Insights and Ideas

  • The "Haystack" Penalty: The most important "aha" moment is that in high dimensions, the penalty for model complexity must grow with the size of the feature space ($p$), not just the sample size ($n$). $GBIC_p$ uses a $2(\log p^*)$ factor that makes it much more "conservative" and selective than traditional BIC.
  • Misspecification is Estimable: You don't need to know the "true" model to correct for misspecification. The authors proved that the $H_n$ matrix can be consistently estimated using only the observed data and the (wrong) working model.
  • Consistency Matters: The authors provide the first rigorous proofs that these generalized criteria remain "consistent" (they pick the best available model with probability approaching 1) even as the number of variables grows at a polynomial rate relative to the sample size.

Key Results

  • Superior Selection: In simulations where $p=3200$ and $n=100$, $GBIC_p$ recovered the "oracle" working model with 64-77% probability, while traditional AIC and BIC failed completely (0% success rate).
  • Prediction Accuracy: Models selected by $GBIC_p$ consistently showed lower median prediction errors compared to those selected by AIC or BIC, especially as $p$ increased.
  • Parsimony in Real Data: On the Neuroblastoma dataset, $GBIC_p$ selected a highly parsimonious model (only 3 genes) while maintaining competitive classification error rates, suggesting it is better at filtering out high-dimensional noise than its predecessors.

Limitations and Future Work

  • Model Families: The current theory is primarily developed for Generalized Linear Models (GLMs). The authors note that extending this to more complex structures, like additive models or survival models (Cox regression), is a vital next step.
  • Computational Cost: While the criteria are easy to calculate once a model is fit, the paper relies on "SICA" (a regularization method) to generate candidate models. Integrating $GBIC_p$ more deeply into the optimization path of high-dimensional solvers could be an interesting engineering challenge.
  • Optimality: While the paper establishes consistency, it leaves open the question of "minimax optimality"—exactly how much penalty is mathematically ideal for a given level of misspecification.

Summary updated

Abstract

Model selection is indispensable to high-dimensional sparse modeling in selecting the best set of covariates among a sequence of candidate models. Most existing work assumes implicitly that the model is correctly specified or of fixed dimensions. Yet model misspecification and high dimensionality are common in real applications. In this paper, we investigate two classical Kullback-Leibler divergence and Bayesian principles of model selection in the setting of high-dimensional misspecified models. Asymptotic expansions of these principles reveal that the effect of model misspecification is crucial and should be taken into account, leading to the generalized AIC and generalized BIC in high dimensions. With a natural choice of prior probabilities, we suggest the generalized BIC with prior probability which involves a logarithmic factor of the dimensionality in penalizing model complexity. We further establish the consistency of the covariance contrast matrix estimator in a general setting. Our results and new method are supported by numerical studies.
BibTeX
@misc{basu2014model,
  author = {Pallavi Basu and Yang Feng and Jinchi Lv},
  title = {{Model Selection in High-Dimensional Misspecified Models}},
  year = {2014},
  eprint = {1412.7468},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1412.7468},
}
Figure