Adaptive Discrete Smoothing for High-Dimensional and Nonlinear Panel Data

Xi Chen, Victor Chernozhukov, Ye Luo, Martin SpindlerarXiv 2019 · arXiv

Introduction

In modern data science, researchers often analyze panel data—tracking multiple individuals, such as taxi drivers or web users, over time. A fundamental challenge in this domain is balancing individual-specific behavior with the need for statistical power. Traditional approaches often fall into two extremes: assuming a homogeneous model for all individuals or estimating entirely separate models for each. The former ignores critical heterogeneity, while the latter suffers from the "curse of dimensionality" when the number of observations per individual is small.

The Adaptive Discrete Smoothing (ADS) Framework

To address these limitations, this paper introduces Adaptive Discrete Smoothing (ADS). Unlike "hard clustering" methods that force individuals into rigid groups—a process that is often computationally expensive and requires pre-specifying the number of clusters—ADS employs a "soft-clustering" approach. It treats clustering as a continuous weighting problem.

The ADS algorithm operates in three distinct stages:

  1. Initial Estimation: A "naive" model is estimated for each individual using only their local data (e.g., via OLS or Lasso).
  2. Similarity Weighting: Similarity scores $W(i, j)$ are calculated between all pairs of individuals based on their initial estimates using an exponential kernel. High weights are assigned to similar individuals, while weights for dissimilar ones decay toward zero.
  3. Weighted Estimation: The final model for each individual is re-estimated by incorporating data from the entire population, weighted by the scores from Stage 2.

This mechanism allows the model to "borrow" statistical strength from "neighbors" in the functional space, effectively reducing variance without the rigid constraints of traditional clustering.

Empirical Evaluation and Results

The effectiveness of ADS was evaluated through both extensive simulations and a real-world application using data from Didi Chuxing (DIDI), China’s leading ride-sharing platform.

Simulation Studies

The authors tested the model against four different Data Generating Processes (DGPs) characterized by varying correlation structures, sparsity, and types of heterogeneity (discrete vs. continuous). These simulations highlight how the method adapts to different underlying data structures.

Performance trends across the four distinct Data Generating Processes (DGPs) evaluated in the study.

Real-World Application: Didi Chuxing

The empirical study focused on predicting the "supply-demand gap" across 66 districts using 46 covariates, including weather and traffic conditions. The ADS-Lasso approach demonstrated superior performance compared to standard benchmarks.

Method Mean Squared Error (MSE) Improvement over OLS
OLS (Standard) 2.25 -
Naive Estimator 1.37 39.1%
Lasso 1.19 47.1%
ADS Lasso (Proposed) 1.00 55.6%

The ADS Lasso outperformed the standard Lasso by approximately 16%, highlighting its ability to leverage cross-sectional similarities to improve individual-level predictions.

Discussion and Future Directions

The primary advantage of ADS lies in its ability to navigate the bias-variance trade-off. By borrowing data from similar individuals, it introduces a small amount of bias but significantly reduces variance, particularly in "small $T$, large $N$" scenarios common in internet and sensor data. Furthermore, the algorithm is highly parallelizable and can be integrated with various base estimators, including Lasso, Ridge, and Neural Networks.

Future work will focus on establishing formal convergence rates and developing automated methods for selecting optimal tuning parameters for the weighting kernel to further enhance the model's robustness and ease of use.

Summary updated

Abstract

In this paper we develop a data-driven smoothing technique for high-dimensional and non-linear panel data models. We allow for individual specific (non-linear) functions and estimation with econometric or machine learning methods by using weighted observations from other individuals. The weights are determined by a data-driven way and depend on the similarity between the corresponding functions and are measured based on initial estimates. The key feature of such a procedure is that it clusters individuals based on the distance / similarity between them, estimated in a first stage. Our estimation method can be combined with various statistical estimation procedures, in particular modern machine learning methods which are in particular fruitful in the high-dimensional case and with complex, heterogeneous data. The approach can be interpreted as a \textquotedblleft soft-clustering\textquotedblright\ in comparison to traditional\textquotedblleft\ hard clustering\textquotedblright that assigns each individual to exactly one group. We conduct a simulation study which shows that the prediction can be greatly improved by using our estimator. Finally, we analyze a big data set from didichuxing.com, a leading company in transportation industry, to analyze and predict the gap between supply and demand based on a large set of covariates. Our estimator clearly performs much better in out-of-sample prediction compared to existing linear panel data estimators.
BibTeX
@misc{chen2019adaptive,
  author = {Xi Chen and Victor Chernozhukov and Ye Luo and Martin Spindler},
  title = {{Adaptive Discrete Smoothing for High-Dimensional and Nonlinear Panel Data}},
  year = {2019},
  eprint = {1912.12867},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.12867},
}
Figure