Abstract
We propose an online debiased lasso (ODL) method for statistical inference in high-dimensional linear models with streaming data. The proposed ODL consists of an efficient computational algorithm for streaming data and approximately normal estimators for the regression coefficients. Its implementation only requires the availability of the current data batch in the data stream and sufficient statistics of the historical data at each stage of the analysis. A dynamic procedure is developed to select and update the tuning parameters upon the arrival of each new data batch so that we can adjust the amount of regularization adaptively along the data stream. The asymptotic normality of the ODL estimator is established under the conditions similar to those in an offline setting and mild conditions on the size of data batches in the stream, which provides theoretical justification for the proposed online statistical inference procedure. We conduct extensive numerical experiments to evaluate the performance of ODL. These experiments demonstrate the effectiveness of our algorithm and support the theoretical results. An air quality dataset and an index fund dataset from Hong Kong Stock Exchange are analyzed to illustrate the application of the proposed method.
Introduction
In the modern landscape of data science, researchers frequently encounter two simultaneous challenges: high dimensionality, where the number of features ($p$) significantly exceeds the number of observations ($n$), and streaming arrival, where data arrives in continuous batches and is too massive to store in memory.
While the Lasso (Least Absolute Shrinkage and Selection Operator) is the standard tool for high-dimensional regression, it is inherently biased because it shrinks coefficients toward zero to achieve sparsity. In offline settings, "debiased Lasso" techniques have been developed to recover the statistical distribution of estimators, allowing for hypothesis testing. However, existing online algorithms focus almost entirely on optimization—finding point estimates—rather than inference. This paper introduces the Online Debiased Lasso (ODL), a framework that enables real-time statistical inference, allowing researchers to construct valid confidence intervals without re-accessing historical raw data.
Methodology
The ODL framework utilizes a three-step procedure that relies exclusively on the current data batch and the sufficient statistics (summary matrices) of the past.
1. Online Lasso Estimation
Traditional coordinate descent requires the full dataset to be available. Instead, ODL employs a combination of gradient descent and soft thresholding. The gradient is calculated using summary statistics—specifically $X^TX$ and $X^Ty$. These matrices have fixed dimensions ($p \times p$ and $p \times 1$, respectively), meaning the memory footprint does not grow with the number of observations.
2. Online Low-Dimensional Projection
To "debias" the Lasso estimator, the method projects each feature onto the space spanned by other features. ODL adapts this projection into an online format, updating the projection matrix as new batches arrive to ensure the estimator remains approximately normal.
3. Adaptive Tuning
A significant hurdle in online learning is selecting the regularization parameter ($\lambda$). ODL implements a "rolling-original-recalibration" strategy. The model uses the previous state to predict the newly arrived batch; the $\lambda$ that minimizes this prediction error is selected for the subsequent update, allowing the model to adjust its sparsity level dynamically.
Key Insights
- Sufficient Statistics: The core innovation is the realization that raw history is unnecessary for debiasing. By maintaining $p \times p$ matrices, ODL captures all necessary information from millions of previous observations.
- Real-Time Inference: Unlike "black box" machine learning models, ODL provides a t-statistic for every feature in real-time. This allows researchers to quantify uncertainty and establish confidence levels for specific variables as the stream evolves.
- Asymptotic Normality: The authors provide mathematical proof that the online estimator behaves like a normal distribution as data accumulates, providing the theoretical foundation for valid p-values and confidence intervals.
Empirical Results
The ODL method was evaluated through both simulations and real-world datasets:
- Simulated Data: In high-dimensional scenarios (e.g., $p=1000$), Q-Q plots confirmed that the estimators followed the predicted normal distribution, validating the theoretical claims.
- Beijing PM2.5 Analysis: Using an hourly air quality dataset, ODL identified the influence of meteorological features on pollution. It successfully captured that Southeast winds have a positive influence on pollution levels, while Northwest winds have a negative impact, aligning with established environmental science.
- Hang Seng Index (HSI) Fund Data: In an analysis of 1,148 stocks, ODL produced significantly tighter confidence intervals than standard Ordinary Least Squares (OLS). In several instances, OLS bias was found to be ten times higher than that of ODL, demonstrating the latter's superior precision in high-dimensional financial tracking.
Limitations and Future Work
While ODL offers a robust solution for streaming inference, it faces a memory bottleneck if the number of features $p$ is extremely large (e.g., millions), as it requires storing a $p \times p$ matrix. Future work may explore Eigenvalue Decomposition (EVD) to compress these matrices. Additionally, the current framework is designed for linear models; extending ODL to Generalized Linear Models (GLMs) and addressing the recovery of variables mistakenly excluded in early streaming stages remain open areas for research.
Summary updated