Do We Exploit all Information for Counterfactual Analysis? Benefits of Factor Models and Idiosyncratic Correction

Jianqing Fan, Ricardo P. Masini, Marcelo C. Medeiros

Here is a concise, engaging summary of the paper, tailored for a technically literate audience.

TL;DR

The paper introduces FarmTreat (Factor-Adjusted Regularized Method for Treatment evaluation), a novel high-dimensional statistical framework that combines factor models with sparse regression to construct highly accurate counterfactuals. When applied to randomized pricing experiments from a major Brazilian retailer, the method successfully handles complex cross-sectional dependencies and reveals massive geographic heterogeneity in consumer price elasticity, showing that optimal prices are often lower than current retail rates.

What Problem Does This Paper Address?

Determining the optimal price of a product to maximize profit or revenue requires an accurate estimate of price elasticity. However, standard regression techniques usually fail due to confounding factors and price endogeneity. While randomized controlled trials (RCTs) help isolate the effect of price changes, standard evaluation methods like Difference-in-Differences (DiD) break down when the parallel trends hypothesis is violated due to strong local variations in consumer behavior.

Alternative approaches, such as Synthetic Control (SC) methods, construct artificial counterfactuals from untreated peer units. Yet, in modern high-dimensional settings where data is collected across hundreds of regions, these methods face severe technical hurdles: the data is often non-stationary, and the high correlation (cross-dependence) between regions violates the sparsity assumptions required for standard regularization techniques (like LASSO) to work effectively.

Methods

The authors propose FarmTreat, a four-step estimation procedure designed to exploit all available information in a high-dimensional panel dataset without imposing strict, unrealistic sparsity constraints on the raw data:

  1. De-trending and De-seasonalization: The effects of known exogenous variables (e.g., day-of-the-week dummies, deterministic trends, calendar effects) are removed from both the treated and peer units using a first-stage regression.
  2. Factor Extraction: A latent factor model is fitted to the residuals of the first step to capture the common, unobserved economic shocks or seasonal dynamics driving all units simultaneously.
  3. Idiosyncratic Regularization (The "Lifting" Step): The remaining residuals—the idiosyncratic components—are isolated. Because the common factors have been stripped away, these components are much less correlated. A sparse LASSO regression is then used to map the idiosyncratic relationships between the treated unit and its peers.
  4. Counterfactual Projection: The combined factor and sparse idiosyncratic models are projected into the post-intervention period to predict what would have happened to the treated unit without the intervention, allowing for precise treatment effect estimation and inference.

Datasets

The methodology is validated using two distinct data environments:

  • Synthetic Simulations: Monte Carlo simulations modeling a data-generating process with two latent factors and varying degrees of sparse cross-dependence among idiosyncratic components across different sample sizes ($T_0$ ranging from 50 to 1000).
  • Real-World Retail Dataset: A novel, proprietary dataset consisting of daily sales and prices of five distinct products in the Sweets and Candies category from a major retailer in Brazil. The data spans 2016 and 2017, covering more than 400 municipalities containing over 1,400 stores.

Key Insights and Ideas

  • The Power of "Lifting": Standard high-dimensional methods fail when predictors are highly correlated because variable selection becomes inconsistent. FarmTreat avoids this by using factor models to "lift" or decorrelate the data, making the subsequent LASSO step highly effective and mathematically robust.
  • Mutual Rescue of Factors and Sparsity: Pure factor models (like Principal Component Regression) perform poorly when the sample size is small because latent factors cannot be precisely estimated. FarmTreat's sparse regression step on the idiosyncratic components acts as a safety net, capturing localized dependencies that the factor model misses.
  • Locality Matters in Retail: The paper provides strong empirical proof that price elasticities are highly localized. A single national or state-level pricing strategy ignores massive municipal-level heterogeneity, leading to lost revenue.

Key Results

  • Simulation Superiority: In finite-sample simulations, FarmTreat consistently achieved lower Mean Squared Error (MSE) than both pure sparse regression (ArCo) and pure factor models (PCR), particularly in small samples.
  • Empirical Impact: In the retail application, price changes yielded statistically significant demand shifts in more than 20% of the treated municipalities.
  • Pricing Strategy Shift: For the majority of municipalities where price changes had a significant impact, the model revealed that the retailer's current prices were too high. The optimal policy for profit maximization actually required decreasing prices, contradicting simple before-and-after estimators which tended to overestimate sales resilience.

Limitations and Future Work

  • Inventory Shock Vulnerability: The model struggled significantly with one specific product (Product V), yielding a very low $R^2$. The authors traced this to severe supply chain stockouts during the experiment period, demonstrating that unmodeled physical constraints (like zero product availability) can break demand estimation.
  • Asymmetric Sample Requirements: The inferential framework relies on having a pre-intervention period that is substantially longer than the post-intervention period ($T_0 \gg T_2$).
  • Unresolved Non-Significance: In roughly 80% of the municipalities, the treatment effects were not statistically significant. The authors note that further iterative experiments are required to determine whether these areas are truly price-inelastic or if the signal was simply drowned out by local noise.

Summary updated

Abstract

Optimal pricing, i.e., determining the price level that maximizes profit or revenue of a given product, is a vital task for the retail industry. To select such a quantity, one needs first to estimate the price elasticity from the product demand. Regression methods usually fail to recover such elasticities due to confounding effects and price endogeneity. Therefore, randomized experiments are typically required. However, elasticities can be highly heterogeneous depending on the location of stores, for example. As the randomization frequently occurs at the municipal level, standard difference-in-differences methods may also fail. Possible solutions are based on methodologies to measure the effects of treatments on a single (or just a few) treated unit(s) based on counterfactuals constructed from artificial controls. For example, for each city in the treatment group, a counterfactual may be constructed from the untreated locations. In this paper, we apply a novel high-dimensional statistical method to measure the effects of price changes on daily sales from a major retailer in Brazil. The proposed methodology combines principal components (factors) and sparse regressions, resulting in a method called Factor-Adjusted Regularized Method for Treatment evaluation (\texttt{FarmTreat}). The data consist of daily sales and prices of five different products over more than 400 municipalities. The products considered belong to the \emph{sweet and candies} category and experiments have been conducted over the years of 2016 and 2017. Our results confirm the hypothesis of a high degree of heterogeneity yielding very different pricing strategies over distinct municipalities.
BibTeX
@misc{fan2020do,
  author = {Jianqing Fan and Ricardo P. Masini and Marcelo C. Medeiros},
  title = {{Do We Exploit all Information for Counterfactual Analysis? Benefits of Factor Models and Idiosyncratic Correction}},
  year = {2020},
  eprint = {2011.03996},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.03996},
}
Figure