An Asympirical Smoothing Parameters Selection Approach for Smoothing Spline ANOVA Models in Large Samples

Xiaoxiao Sun, Wenxuan Zhong, Ping MaarXiv 2020 · DOI · arXiv

Abstract

Large datasets generated from modern scientific and industrial sources present significant challenges for classical statistical models. Smoothing Spline ANOVA (SS-ANOVA) models, while powerful for non-parametric regression and modeling complex interactions, are often computationally prohibitive due to the $O(n^3)$ complexity of selecting smoothing parameters via Generalized Cross-Validation (GCV). This paper presents an "asympirical" approach—combining asymptotic analysis with empirical subsampling—to select smoothing parameters efficiently. By leveraging the theoretical power-law relationship between optimal smoothing parameters and sample size, the proposed method reduces computational costs to $O(b^3)$, where $b$ is a small subsample. We demonstrate that this method maintains optimal convergence rates and provides superior predictive performance on datasets with nearly one million observations.

Introduction

Smoothing Spline ANOVA (SS-ANOVA) models provide a flexible framework for multivariate function estimation, allowing researchers to decompose complex functions into main effects and interactions. However, the utility of SS-ANOVA is frequently limited by the computational burden of smoothing parameter selection. The standard approach, GCV, requires the repeated inversion of large matrices, a process that scales cubically with the number of observations ($n$). In the era of big data, where $n$ can exceed $10^6$, standard GCV becomes mathematically and hardware-infeasible.

The Asympirical Methodology

The "asympirical" method addresses this bottleneck by bridging the gap between asymptotic theory and empirical practice. The approach is structured in two primary phases:

1. Theoretical Foundation (Asymptotic)

Statistical theory indicates that the optimal smoothing parameter $\lambda$ for an SS-ANOVA model follows a predictable decay rate relative to the sample size $n$. Specifically, the relationship can be modeled as: $$\lambda \approx C n^{-r/(pr+1)}$$ In this expression, $C$ is an unknown constant, while $r$ and $p$ are parameters related to the smoothness of the underlying functional space. This power-law relationship suggests that if we can determine the constant $C$, we can calculate the optimal $\lambda$ for any sample size.

2. Subsample Extrapolation (Empirical)

To estimate the unknown constant $C$ without processing the full dataset, the researchers employ a subsampling strategy:

  • Subsampling: A small random subsample of size $b$ is drawn from the full dataset.
  • Local Optimization: Standard GCV is performed on this small subsample to find the optimal $\lambda(b)$. Because $b \ll n$, the $O(b^3)$ cost is negligible.
  • Extrapolation: Using the theoretical bridge, the smoothing parameter for the full dataset is calculated as: $$\lambda_{ASP}(n) = \lambda_{GCV}(b) \cdot (n/b)^{-r/(pr+1)}$$

By performing the computationally expensive GCV only on the subsample, the method bypasses the $O(n^3)$ bottleneck entirely, effectively scaling the model to massive datasets.

Empirical Validation

The authors validated the asympirical approach through extensive simulations and two high-dimensional real-world applications.

Superconductivity Data

Using a dataset of 21,263 observations with 42 predictors to predict the critical temperature of superconductors, the asympirical method demonstrated a dramatic increase in efficiency. It achieved a processing time of 0.03 seconds, compared to 40.56 seconds for standard GCV—a speedup of approximately 1,350 times—while maintaining comparable accuracy.

Molecular Dynamics Data

The method's scalability was tested on a massive molecular dynamics dataset containing 893,238 observations of malondialdehyde trajectories. With 36 predictors representing atomic distances, standard GCV was computationally infeasible. The asympirical method, however, completed the parameter selection in under 2 seconds. Furthermore, it outperformed competing fast methods, such as "Skip" and "BAM" (Fast REML), in terms of Root Mean Squared Error (RMSE), proving that computational speed does not have to come at the cost of predictive precision.

Key Insights and Theoretical Guarantees

Beyond speed, the asympirical method offers several critical advantages:

  • Optimality: The authors prove that the extrapolated smoothing parameters achieve the optimal convergence rate. This ensures that as the dataset grows, the model's accuracy improves at the same rate as it would under full GCV.
  • Interaction Modeling: Unlike some simplified additive models that ignore complex interactions to save time, this approach allows SS-ANOVA to retain its full capacity for modeling non-additive interactions.
  • Stability: To mitigate noise in the subsample, the authors recommend "asymptotic sampling," which involves averaging results from multiple subsamples to stabilize the estimation of the constant $C$.

Limitations and Future Directions

While the method is robust, its performance depends on the selection of the smoothness parameters $r$ and $p$, and the subsample size $b$. If $b$ is too small, the initial GCV estimate may be unstable. Current research focuses on Gaussian noise; future work could extend this asympirical logic to the broader Exponential Family of distributions, such as Binomial or Poisson models, further expanding the reach of SS-ANOVA in big data contexts.

Conclusion

The asympirical approach represents a significant step forward in making non-parametric ANOVA models practical for modern data science. By substituting cubic computational growth with a theoretical extrapolation, it enables the analysis of datasets with millions of points in seconds, providing a scalable solution for high-dimensional regression challenges.

Summary updated

Abstract

Large samples have been generated routinely from various sources. Classic statistical models, such as smoothing spline ANOVA models, are not well equipped to analyze such large samples due to expensive computational costs. In particular, the daunting computational costs of selecting smoothing parameters render smoothing spline ANOVA models impractical. In this article, we develop an asympirical, i.e., asymptotic and empirical, smoothing parameters selection approach for smoothing spline ANOVA models in large samples. The idea of this approach is to use asymptotic analysis to show that the optimal smoothing parameter is a polynomial function of the sample size and an unknown constant. The unknown constant is then estimated through empirical subsample extrapolation. The proposed method significantly reduces the computational costs of selecting smoothing parameters in high-dimensional and large samples. We show smoothing parameters chosen by the proposed method tend to the optimal smoothing parameters that minimise a specific risk function. In addition, the estimator based on the proposed smoothing parameters achieves the optimal convergence rate. Extensive simulation studies demonstrate the numerical advantage of the proposed method over competing methods in terms of relative efficacies and running time. On an application to molecular dynamics data with nearly one million observations, the proposed method has the best prediction performance.
BibTeX
@misc{sun2020asympirical,
  author = {Xiaoxiao Sun and Wenxuan Zhong and Ping Ma},
  title = {{An Asympirical Smoothing Parameters Selection Approach for Smoothing Spline ANOVA Models in Large Samples}},
  year = {2020},
  eprint = {2004.10271},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.10271},
}
Figure