Individual Shrinkage for Random Effects

Raffaella Giacomini, Sokbae Lee, Silvia SarpietroarXiv 2023 · DOI · arXiv

TL;DR

This paper introduces a novel "Individual Weights" (IW) shrinkage method for random effects estimation that prioritizes the accuracy of individual-level forecasts over aggregate group performance. By leveraging an individual’s own history rather than cross-sectional averages, the approach avoids the "tyranny of the majority"—a phenomenon where standard estimators produce large biases for outliers—and provides a robust framework for decision-making in short-panel data.

What Problem Does This Paper Address?

In econometrics, researchers often work with "micropanels"—datasets with many individuals ($N$) but very few time observations ($T$). To handle the noise in such short time series, standard practice uses shrinkage estimators like James-Stein (JS) or Empirical Bayes (EB). These methods "borrow strength" from the entire group, pulling individual estimates toward a global mean.

The catch? These methods are designed to minimize aggregate loss. If you are a policy-maker deciding whether to dismiss a specific teacher based on "value-added" scores or a bank providing personalized financial advice, aggregate accuracy doesn't matter as much as being right about that specific individual. The authors argue that traditional shrinkage suffers from the "tyranny of the majority": it forces outliers to look like the average, leading to significant bias and poor decisions for the very individuals who stand out most.

Methods

The authors propose a class of Shrinkage with Individual Weights (IW). Unlike JS or EB, which use cross-sectional information to determine how much to shrink, IW uses the individual’s own past history.

Key technical components include:

  • Semiparametric Approach: Unlike Empirical Bayes, which requires strict distributional assumptions (like normality) and "exchangeability" (the idea that all individuals are drawn from the same distribution), IW is more flexible and robust to misspecification.
  • Minimax Regret (IW-MR): Because time series are short, estimating "perfect" (oracle) weights is nearly impossible. The authors use a Minimax Regret criterion. This decision-theoretic framework identifies weights that perform well across the entire unknown parameter space, minimizing the "regret" (the extra loss incurred) of not knowing the true underlying parameters.
  • Feasible Weights: They derive practical formulas for weights (IW-MR, IW-O, and IW-MSFE) that can be calculated directly from short-sample data without needing a large $N$.

Datasets

The paper evaluates the method using both simulations and real-world economic data:

  1. Monte Carlo Simulations: Used to test the method against Normal, Laplace, and heavy-tailed Double-Pareto distributions.
  2. Firm Discrimination Data: A revisit of the Kline et al. (2022) study, which sent fictitious job applications to 108 large U.S. employers to measure gender discrimination (72 firms over 5 waves).
  3. Panel Study of Income Dynamics (PSID): Earnings data for male workers from 1968–1993, used to forecast individual income residuals.

Key Insights and Ideas

  • Individual Accuracy vs. Aggregate Performance: The most important "aha moment" is that an estimator can be "optimal" for a group while being "terrible" for the most interesting individuals in that group (the outliers).
  • The Power of Heavy Tails: IW is particularly advantageous when the underlying distribution has heavy tails. In these cases, the "majority" is a poor proxy for the "outliers," and IW’s reliance on individual history allows it to capture these extreme values more accurately.
  • Robustness to Sample Composition: Because IW doesn't rely on the cross-section to determine shrinkage for a specific person, your estimate for "Person A" doesn't change just because you added "Person B" to your dataset. This provides better "external validity" than Empirical Bayes.

Key Results

  • Overcoming Bias: In simulations, the IW-MR estimator consistently outperformed the time-series mean and the pooled mean in terms of Minimax Regret, especially when the "signal-to-noise" ratio was uncertain.
  • Policy Implications in Discrimination: When re-analyzing firm discrimination, IW-MR identified a higher probability of firms being "highly discriminatory" (4.29%) compared to the Empirical Bayes approach (1.43%). This suggests that standard EB might be "shrinking away" evidence of systemic discrimination by forcing firms toward the mean.
  • Earnings Forecasting: In the PSID application, IW-MR achieved better aggregate accuracy than both the time-series mean and the pooled mean, proving that targeting individual accuracy can sometimes improve group-level results as well.

Limitations and Future Work

  • Common Mean Assumption: The current framework assumes all individuals are shrunk toward a single common mean (though the authors suggest this can be handled by demeaning data or using group-specific means).
  • Linear Models: The derivation currently focuses on linear models and residuals. The authors note that extending this to models with heterogeneous coefficients for covariates is a logical next step but more complex.
  • Hybrid Estimators: The conclusion suggests that a "best of both worlds" approach—combining individual-level shrinkage with James-Stein cross-sectional information—could be a fruitful area for future research.

Summary updated

Abstract

This paper develops a novel approach to random effects estimation and individual-level forecasting in micropanels, targeting individual accuracy rather than aggregate performance. The conventional shrinkage methods used in the literature, such as the James-Stein estimator and Empirical Bayes, target aggregate performance and can lead to inaccurate decisions at the individual level. We propose a class of shrinkage estimators with individual weights (IW) that leverage an individual's own past history, instead of the cross-sectional dimension. This approach overcomes the "tyranny of the majority" inherent in existing methods, while relying on weaker assumptions. A key contribution is addressing the challenge of obtaining feasible weights from short time-series data and under parameter heterogeneity. We discuss the theoretical optimality of IW and recommend using feasible weights determined through a Minimax Regret analysis in practice.
BibTeX
@misc{giacomini2023individual,
  author = {Raffaella Giacomini and Sokbae Lee and Silvia Sarpietro},
  title = {{Individual Shrinkage for Random Effects}},
  year = {2023},
  eprint = {2308.01596},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.01596},
}
Figure