Estimation of High-Dimensional Nonlinear Vector Autoregressive Models

Yuefeng Han, Likai Chen, Wei Biao WuIEEE Trans. Information Theory 2025 · OpenReview

TL;DR

This paper introduces a new framework for analyzing complex, high-dimensional time series data by extending traditional linear models to capture nonlinear dynamics. The authors develop a "sparse additive nonlinear VAR model" and provide theoretical guarantees for its performance, particularly by deriving novel, sharp inequalities for dependent data. This work is crucial for fields like genomics and economics where complex, nonlinear interactions are common but hard to model.

What Problem Does This Paper Address?

Many real-world systems, from economic markets to biological networks, are characterized by a large number of interacting variables whose dynamics evolve over time. Traditional "Vector Autoregressive" (VAR) models are powerful tools for capturing these temporal dependencies. However, most existing VAR models assume linear relationships between variables. This is a significant limitation because many natural processes exhibit complex, nonlinear interactions. For example, in gene regulatory networks, the expression of one gene might not just linearly influence another, but through a more intricate, nonlinear mechanism. This paper aims to bridge this gap by developing a flexible framework that can effectively model these high-dimensional, nonlinear time series, going beyond the restrictive assumptions of linearity.

Methods

The core of this paper lies in proposing a sparse additive nonlinear VAR model. Instead of a simple linear combination of past values, the model assumes that the influence of past variables on the current state can be represented as a sum of simpler, nonlinear functions of individual past variables. This "additive" structure simplifies the problem. Crucially, the authors employ basis expansions to represent these nonlinear functions, allowing them to leverage powerful tools from sparse modeling.

Key methodological contributions include:

  • Non-parametric approach: The model doesn't assume a specific functional form for the nonlinear relationships, making it more adaptable.
  • Sparsity: The framework encourages sparsity, meaning it aims to identify only the most influential direct relationships, which is essential in high-dimensional settings where most interactions are likely weak or non-existent. This is achieved through an $l_1$-regularized optimization problem (similar to Lasso).
  • Sharp Bernstein-type inequalities: A major theoretical contribution is the derivation of new, sharp inequalities for tail probabilities of dependent processes. These inequalities are crucial for analyzing the statistical properties of their proposed estimator and are shown to be as sharp as classical Bernstein inequalities for independent data, a significant improvement over existing methods for dependent data.

Datasets

The paper uses a real biological gene regulatory network time series dataset from the E. coli SOS DNA repair system. This dataset consists of expression levels of 8 genes measured at 50 time points. The authors compare their proposed method against a Lasso-regularized linear VAR model on this dataset.

Key Insights and Ideas

  • Nonlinearity is crucial: The paper strongly suggests that linear models are insufficient for many real-world time series problems, particularly in biological systems, and that capturing nonlinear dynamics is essential for accurate modeling and inference.
  • Additive structure simplifies complexity: By decomposing complex nonlinear interactions into sums of simpler, univariate nonlinear functions, the problem becomes more tractable without sacrificing too much modeling power.
  • Sharp inequalities unlock theoretical guarantees: The development of sharp Bernstein-type inequalities for dependent, non-Gaussian data is a significant theoretical advancement. These inequalities are the backbone for proving the theoretical properties (like convergence rates and model selection consistency) of their proposed sparse nonlinear VAR estimator.
  • Decoupling smoothness and sparsity: The proposed framework effectively separates the problem of estimating the smoothness of nonlinear functions from identifying which variables are truly influential, leading to efficient algorithms.

Key Results

  • Theoretical Guarantees: The paper provides theoretical guarantees for the proposed $l_1$-regularized estimator, including convergence rates and model selection consistency, under various conditions (e.g., non-Gaussian errors, different moment conditions).
  • Improved Performance on Real Data: In the analysis of the E. coli SOS DNA repair network, their nonlinear VAR model outperformed a standard linear VAR model in terms of AUROC and AUPR, demonstrating its ability to capture nonlinear dynamics and identify more true gene regulatory interactions.
  • Simulation Studies: Extensive simulations showed that the proposed method performs well across different network structures ("random," "band," "cluster") and various data dimensions and sample sizes, with performance generally improving as sample size increases and degrading gracefully as dimension increases.

Limitations and Future Work

  • Computational Complexity: While the method is designed to scale to high dimensions, the computational cost can still be significant, especially with a large number of basis functions or a very high-dimensional state space.
  • Tuning Parameter Sensitivity: The performance of $l_1$-regularized methods often depends on the choice of tuning parameters, and the paper acknowledges that the truncation level of basis expansions can influence convergence rates.
  • Future Work: The authors suggest that their framework and derived inequalities can be useful for other high-dimensional linear and nonlinear VAR problems. Further research could explore adaptive selection of basis functions, more sophisticated dependence measures, and applications to other complex systems.

Summary updated

Abstract

High-dimensional vector autoregressive (VAR) models have numerous applications in fields such as econometrics, biology, climatology, among others. While prior research has mainly focused on linear VAR models, these approaches can be restrictive in practice. To address this, we introduce a high-dimensional non-parametric sparse additive model, providing a more flexible framework. Our method employs basis expansions to construct high-dimensional nonlinear VAR models. We derive convergence rates and model selection consistency for least squared estimators, considering dependence measures of the processes, error moment conditions, sparsity, and basis expansions. Our theory significantly extends prior linear VAR models by incorporating both non-Gaussianity and non-linearity. As a key contribution, we derive sharp Bernstein-type inequalities for tail probabilities in both non-sub-Gaussian linear and nonlinear VAR processes, which match the classical Bernstein inequality for independent random variables. Additionally, we present numerical experiments that support our theoretical findings and demonstrate the advantages of the nonlinear VAR model for a gene expression time series dataset.
BibTeX
@article{han2025estimation,
  author = {Yuefeng Han and Likai Chen and Wei Biao Wu},
  title = {{Estimation of High-Dimensional Nonlinear Vector Autoregressive Models}},
  journal = {IEEE Transactions on Information Theory},
  year = {2025},
  url = {https://openreview.net/forum?id=b87j7pBtvn},
}
Figure