Statistical Inference in High-dimensional Generalized Linear Models with Streaming Data

Lan Luo, Ruijian Han, Yuanyuan Lin, Jian HuangarXiv 2021 · DOI · arXiv

Abstract

In this paper we develop an online statistical inference approach for high-dimensional generalized linear models with streaming data for real-time estimation and inference. We propose an online debiased lasso (ODL) method to accommodate the special structure of streaming data. ODL differs from offline debiased lasso in two important aspects. First, in computing the estimate at the current stage, it only uses summary statistics of the historical data. Second, in addition to debiasing an online lasso estimator, ODL corrects an approximation error term arising from nonlinear online updating with streaming data. We show that the proposed online debiased estimators for the GLMs are consistent and asymptotically normal. This result provides a theoretical basis for carrying out real-time interim statistical inference with streaming data. Extensive numerical experiments are conducted to evaluate the performance of the proposed ODL method. These experiments demonstrate the effectiveness of our algorithm and support the theoretical results. A streaming dataset from the National Automotive Sampling System-Crashworthiness Data System is analyzed to illustrate the application of the proposed method.

Introduction

In the contemporary landscape of "Big Data," information is increasingly generated at high velocity through sensor networks, financial transactions, and genomic monitoring. Analyzing these data streams presents two significant challenges: high dimensionality and memory constraints. Often, the number of potential predictors ($p$) is very large, sometimes exceeding the number of observations ($n$) in the early stages of data collection. Furthermore, the sheer volume of data makes it computationally prohibitive to store all historical raw data for model re-estimation every time a new batch arrives.

While the "Debiased Lasso" has become a standard for statistical inference in high-dimensional settings—allowing for the calculation of p-values and confidence intervals—it is fundamentally an offline method. Existing online adaptations frequently struggle with high-dimensional noise or require substantial initial data to achieve stability. This research introduces the Online Debiased Lasso (ODL), a "one-pass" algorithm designed to provide the statistical reliability of offline methods without the associated computational and memory overhead.

Methodology

The ODL framework is built upon three core innovations designed to handle the recursive nature of streaming data within Generalized Linear Models (GLMs):

1. Approximate Summary Statistics

To eliminate the need for storing raw historical data, ODL utilizes a first-order Taylor expansion to approximate the score function (the gradient of the likelihood). This allows the model to update its parameters using only the previous estimate and an aggregated information matrix, effectively summarizing the past without retaining it.

2. Online Error Correction

In non-linear models such as logistic regression, recursive updates can lead to an accumulation of approximation errors over time. The ODL method introduces a specific correction term to cancel out this "drift." This ensures that the final estimator maintains asymptotic normality, which is essential for valid statistical inference.

3. Adaptive Tuning

Traditional penalty selection ($\lambda$) via cross-validation requires access to the full dataset. ODL implements a "rolling-recalibration" approach, where the current data batch serves as a validation set for the model derived from previous batches. This allows for the dynamic selection of the optimal penalty on the fly.

Experimental Evaluation

The researchers validated the ODL method through both synthetic simulations and real-world data analysis.

Synthetic Simulations

Simulations involved high-dimensional data ($p=100$ and $p=600$) following a logistic regression model. These tests evaluated the algorithm's performance across varying signal strengths and correlation structures. The results indicated that ODL achieved a 95% coverage probability for confidence intervals, matching the performance of benchmark offline methods. In contrast, standard Maximum Likelihood Estimation (MLE) proved unstable in these high-dimensional settings.

NASS CDS Dataset Analysis

The method was applied to streaming data from the National Automotive Sampling System-Crashworthiness Data System. This dataset comprised monthly accident reports over seven years (84 batches), involving 2,118 vehicles and 62 predictors, such as driver age, alcohol use, and seatbelt status. ODL successfully identified that the interaction between "Young Age" and "Alcohol" was the most significant predictor of fatal crashes. The analysis also demonstrated an "accrual of evidence," where confidence bands narrowed as more data batches were processed.

Key Insights

  • Computational Efficiency: In comparative simulations, ODL was approximately 16 times faster than its offline counterpart, as it avoids the need to re-process the entire cumulative dataset at each step.
  • Theoretical Rigor: The research proves that the online error correction term is mathematically mandatory for maintaining the "oracle" properties of the estimator in a streaming context.
  • Real-Time Inference: Unlike many online learning algorithms that focus solely on prediction, ODL enables real-time inference, allowing researchers to monitor the evolving significance of variables as data arrives.

Conclusion and Future Work

The Online Debiased Lasso provides a memory-efficient and mathematically sound framework for high-dimensional inference on streaming data. However, the authors note certain limitations that provide avenues for future research. Currently, the model assumes a homogeneous data stream; future iterations will need to address "concept drift," where underlying data patterns change over time. Additionally, exploring robust loss functions, such as Huber’s loss, could improve performance in the presence of outliers. Finally, the authors aim to develop mechanisms to recover significant variables that might be prematurely excluded by the Lasso penalty during early, small-sample batches.

Summary updated

Abstract

In this paper we develop an online statistical inference approach for high-dimensional generalized linear models with streaming data for real-time estimation and inference. We propose an online debiased lasso (ODL) method to accommodate the special structure of streaming data. ODL differs from offline debiased lasso in two important aspects. First, in computing the estimate at the current stage, it only uses summary statistics of the historical data. Second, in addition to debiasing an online lasso estimator, ODL corrects an approximation error term arising from nonlinear online updating with streaming data. We show that the proposed online debiased estimators for the GLMs are consistent and asymptotically normal. This result provides a theoretical basis for carrying out real-time interim statistical inference with streaming data. Extensive numerical experiments are conducted to evaluate the performance of the proposed ODL method. These experiments demonstrate the effectiveness of our algorithm and support the theoretical results. A streaming dataset from the National Automotive Sampling System-Crashworthiness Data System is analyzed to illustrate the application of the proposed method.
BibTeX
@misc{luo2021statistical,
  author = {Lan Luo and Ruijian Han and Yuanyuan Lin and Jian Huang},
  title = {{Statistical Inference in High-dimensional Generalized Linear Models with Streaming Data}},
  year = {2021},
  eprint = {2108.04437},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.04437},
}
Figure