Spectral clustering with variance information for group structure estimation in panel data

Lu Yu, Jiaying Gu, Stanislav VolgushevJournal of Econometrics 2025 · OpenReview

TL;DR

This paper introduces a novel spectral clustering method for identifying group structures in panel data, specifically by incorporating the variance of individual coefficient estimates. This approach improves upon existing methods by accounting for estimation uncertainty, leading to more accurate group identification, especially in complex scenarios.

What Problem Does This Paper Address?

Panel data, common in fields like economics and statistics, often exhibits heterogeneity among individuals. A key challenge is to uncover underlying group structures where individuals within a group share similar responses to observed characteristics, even when these groups are not explicitly known. Traditional methods often pool information or rely on penalized loss functions, but these can be computationally intensive or fail to fully leverage the information contained in the uncertainty of individual estimates. This paper addresses the need for a method that is both statistically robust by accounting for estimation uncertainty and computationally efficient, particularly for large datasets.

Methods

The core contribution is a spectral clustering approach that utilizes a dissimilarity measure incorporating the variance-covariance of individual coefficient estimates. This is a departure from methods that solely rely on point estimates. The authors motivate this by a local analysis showing that variances contain valuable information for group structure estimation.

The proposed method involves:

  1. Estimating individual coefficients and their variance-covariance matrices: This is a standard step in panel data analysis.
  2. Constructing a dissimilarity matrix: This is where the novelty lies. The dissimilarity between individuals is weighted by the inverse of their estimated variance-covariance matrices, effectively giving more importance to estimates with higher precision.
  3. Applying spectral clustering: The dissimilarity matrix is then used as input for a spectral clustering algorithm. This algorithm leverages the eigenvalues and eigenvectors of a graph Laplacian derived from the dissimilarity matrix to identify clusters.
  4. Heuristic for determining the number of groups: The paper also proposes a heuristic based on the eigen-gap of the graph Laplacian to automatically select the optimal number of groups.

The method is designed to be computationally feasible and can be applied even when only parameter estimates and their uncertainty are available, not raw individual-level data.

Datasets

The paper evaluates its method using simulated data across various models and settings. It also applies the methodology to two real-world datasets:

  • Environmental Kuznet Curves (EKC) data: State-level panel data from the US (1929-1994) on SO2 emissions and per capita income.
  • Intergenerational income mobility data: Publicly available region-specific estimates from the US (Chetty and Hendren, 2018).

Key Insights and Ideas

  • Variance is informative: The variance-covariance of individual coefficient estimates is not just noise; it contains crucial information for identifying group structures. Ignoring it can lead to suboptimal performance.
  • Spectral clustering adapts well: Spectral clustering, when combined with a variance-informed dissimilarity measure, demonstrates superior performance compared to other clustering methods like PAM and iterative k-means, especially for non-elliptical cluster shapes.
  • Computational efficiency: The proposed method avoids computationally intensive re-estimation of models on the full dataset at each iteration, making it scalable for large panel data.
  • Flexibility for data availability: The method can be applied even when only summary statistics (parameter estimates and their uncertainty) are available, which is a significant advantage in many real-world scenarios.

Key Results

The paper presents extensive simulation results showing that the proposed spectral clustering method consistently outperforms existing methods (C-LASSO, PAM, iterative k-means, SBSA, ZWZ19) in terms of both perfect matching and average matching accuracy across various models and data generating processes. For instance, in Model 1 of the simulations, the proposed method achieves near-perfect match rates (e.g., 0.999) while other methods lag significantly. The empirical applications also demonstrate the method's ability to uncover meaningful group structures in environmental pollution and income mobility data.

Limitations and Future Work

  • Independence assumption: The current implementation of the dissimilarity measure and its theoretical analysis assumes independence across individuals. The authors acknowledge that dependence is present in many settings and modeling it is an important area for future research.
  • Short and unbalanced panels: The method's performance might be affected by short and highly unbalanced panels where individual-level estimators have poor quality.
  • Group structure in subsets of coefficients: The paper suggests a two-step procedure where coefficients containing group information are identified first, and then grouping is performed on those coefficients, as a potential improvement.
  • Covariate scaling: The authors mention that re-scaling covariates to bring coefficients to a similar scale could be advantageous and warrants further exploration.
  • Time-varying group heterogeneity: Extending the approach to models with time-varying group heterogeneity is suggested as a natural next step.
  • Theoretical validation of group number selection: While a heuristic for selecting the number of groups is proposed, its theoretical validation is left open for future research.

Summary updated

Abstract

Consider a panel data setting where repeated observations on individuals are available. Often it is reasonable to assume that there exist groups of individuals that share similar effects of observed characteristics, but the grouping is typically unknown in advance. We first conduct a local analysis which reveals that the variances of the individual coefficient estimates contain useful information for the estimation of group structure. We then propose a method to estimate unobserved groupings for general panel data models that explicitly accounts for the variance information. Our proposed method remains computationally feasible with a large number of individuals and/or repeated measurements on each individual. The developed ideas can also be applied even when individual-level data are not available and only parameter estimates together with some quantification of estimation uncertainty are given to the researcher. A thorough simulation study demonstrates superior performance of our method than existing methods and we apply the method to two empirical applications. © 2024
BibTeX
@article{yu2025spectral,
  author = {Lu Yu and Jiaying Gu and Stanislav Volgushev},
  title = {{Spectral clustering with variance information for group structure estimation in panel data}},
  journal = {Journal of Econometrics},
  year = {2025},
  url = {https://openreview.net/forum?id=jI3V7I0JZA},
}
Figure