High-Dimensional Bayesian Inference in Nonparametric Additive Models

Zuofeng Shang, Ping Li

Abstract

A fully Bayesian approach is proposed for ultrahigh-dimensional nonparametric additive models in which the number of additive components may be larger than the sample size, though ideally the true model is believed to include only a small number of components. Bayesian approaches can conduct stochastic model search and fulfill flexible parameter estimation by stochastic draws. The theory shows that the proposed model selection method has satisfactory properties. For instance, when the hyperparameter associated with the model prior is correctly specified, the true model has posterior probability approaching one as the sample size goes to infinity; when this hyperparameter is incorrectly specified, the selected model is still acceptable since asymptotically it is proved to be nested in the true model. To enhance model flexibility, two new $g$-priors are proposed and their theoretical performance is examined. We also propose an efficient MCMC algorithm to handle the computational issues. Several simulation examples are provided to demonstrate the computational advantages of our method.

Introduction

In modern statistical analysis, researchers frequently encounter "ultrahigh-dimensional" datasets where the number of potential predictors ($p$) significantly exceeds the number of observations ($n$). In such regimes, traditional regression models fail due to the curse of dimensionality. To maintain interpretability and predictive power, two common assumptions are often employed: sparsity (only a small subset of predictors are truly influential) and additivity (the response is a sum of univariate functions of the predictors).

While frequentist methods like the Lasso or Sure Independence Screening (SIS) have been widely adopted for these problems, they often rely on multi-stage pipelines involving screening, thresholding, and subsequent estimation. Bayesian methods offer a compelling alternative by performing variable selection and function estimation simultaneously within a unified probabilistic framework. However, the theoretical validity of Bayesian nonparametric models in the $p \gg n$ context has historically been difficult to establish. This paper bridges that gap, providing a robust framework for Bayesian inference in nonparametric additive models with proven posterior consistency.

Methodology

Function Representation

The authors represent each unknown component function $f_j$ using a partial Fourier series. By approximating curves through a finite basis of functions, the nonparametric problem is transformed into a high-dimensional parametric problem. This allows the model to capture complex, non-linear relationships without the rigid constraints of linear modeling.

Prior Specifications

To navigate the vast model space, the framework employs several innovative prior structures:

  1. Spike-and-Slab Priors: This mechanism is used to perform variable selection. A "spike" (a point mass at zero) effectively removes non-significant components, while a "slab" (a smooth Gaussian distribution) allows for the estimation of active components.
  2. Size-Control Prior: To prevent overfitting in ultrahigh dimensions, the authors introduce a prior that penalizes or caps the total number of active components ($t_n$). This ensures the model remains sparse and interpretable.
  3. New g-priors: The paper introduces Generalized Zellner-Siow (GZS) and Generalized hyper-g (GHG) priors. These adaptively scale the variance of the Fourier coefficients based on the data, providing greater flexibility than traditional fixed-variance priors.

Computational Implementation

Given the complexity of searching across models of varying dimensions, the authors utilize a Reversible Jump Markov Chain Monte Carlo (RJMCMC) algorithm. This "trans-dimensional" sampler allows the model to move efficiently between different subsets of predictors. A specific "dimension-matching" technique is employed to ensure high acceptance rates and efficient exploration of the posterior distribution.

Theoretical Guarantees

A primary contribution of this work is the rigorous proof of Posterior Model Consistency. The authors demonstrate that if the size-control hyperparameter is correctly specified, the posterior probability of the true model converges to one as the sample size $n$ increases.

Furthermore, the paper establishes Nested Consistency. In scenarios where the maximum model size is misspecified (e.g., set too conservatively), the method is proven to select a model that is nested within the true model. This "graceful failure" ensures that the method remains reliable and avoids selecting spurious noise variables even under sub-optimal hyperparameter settings.

Empirical Results

The proposed Bayesian methods (BGZS and BGHG) were evaluated using simulated datasets designed to mimic challenging high-dimensional scenarios (e.g., $p=1000, n=400$). Comparisons were made against state-of-the-art frequentist approaches, such as the Iterative Nonparametric Independence Screening (INIS-penGAM).

Key findings include:

  • Superior Sparsity: The Bayesian approach consistently produced fewer false positives than frequentist competitors, effectively suppressing irrelevant predictors.
  • Predictive Accuracy: In multiple benchmarks, the Bayesian framework achieved lower Prediction Error (PE), demonstrating that the simultaneous estimation and selection process leads to more robust models.
  • Basis Robustness: The results remained consistent whether using Legendre polynomials or trigonometric bases, suggesting the method is not overly sensitive to the choice of basis functions.

Conclusion

This research provides a theoretically sound and computationally viable path for Bayesian inference in ultrahigh-dimensional nonparametric settings. By combining size-control priors with adaptive $g$-priors, the authors have developed a method that outperforms traditional frequentist pipelines in both selection accuracy and predictive performance. While the MCMC-based approach is more computationally intensive than simple screening methods, its ability to provide a full posterior distribution and guaranteed consistency makes it a powerful tool for complex data analysis. Future work may extend this framework to include interaction terms and non-Gaussian error structures.

Summary updated

Abstract

A fully Bayesian approach is proposed for ultrahigh-dimensional nonparametric additive models in which the number of additive components may be larger than the sample size, though ideally the true model is believed to include only a small number of components. Bayesian approaches can conduct stochastic model search and fulfill flexible parameter estimation by stochastic draws. The theory shows that the proposed model selection method has satisfactory properties. For instance, when the hyperparameter associated with the model prior is correctly specified, the true model has posterior probability approaching one as the sample size goes to infinity; when this hyperparameter is incorrectly specified, the selected model is still acceptable since asymptotically it is proved to be nested in the true model. To enhance model flexibility, two new $g$-priors are proposed and their theoretical performance is examined. We also propose an efficient MCMC algorithm to handle the computational issues. Several simulation examples are provided to demonstrate the computational advantages of our method.
BibTeX
@misc{shang2013high,
  author = {Zuofeng Shang and Ping Li},
  title = {{High-Dimensional Bayesian Inference in Nonparametric Additive Models}},
  year = {2013},
  eprint = {1307.0056},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1307.0056},
}
Figure