Do We Exploit all Information for Counterfactual Analysis? Benefits of Factor Models and Idiosyncratic Correction

Jianqing Fan, Ricardo Masini, Marcelo C. Medeiros

Introduction

In the fields of policy evaluation and retail analytics, the ability to estimate counterfactuals—what would have happened to a specific unit (such as a city, store, or firm) had an intervention not occurred—is critical for decision-making. Whether assessing the impact of a new tax or a price change, researchers rely on identifying a reliable baseline for comparison.

Traditional methods like Difference-in-Differences (DiD) often rely on the "parallel trends" assumption, which posits that treated and control units would follow the same trajectory in the absence of treatment. In complex, high-dimensional environments like retail, this assumption frequently fails. A candy bar's sales in a high-end urban mall rarely mirror those in a rural convenience store. While Synthetic Control (SC) methods attempt to address this by creating a weighted average of peers, they struggle with high dimensionality and non-stationary data. This paper introduces the FarmTreat (Factor-Adjusted Regularized Method for Treatment evaluation) framework to bridge these gaps by leveraging both global market trends and localized idiosyncratic relationships.

The FarmTreat Methodology

FarmTreat is a four-step methodology designed to be robust in environments where data is dense, correlated, and highly heterogeneous. The framework operates by "lifting" the data into a factor structure before applying sparse regression, making the estimation process more efficient.

  1. Heterogeneity Adjustment: The process begins by stripping away known exogenous effects. This includes seasonality (e.g., day-of-the-week effects), deterministic trends, and outliers using standard regression techniques to isolate the core signal.
  2. Factor Modeling (The Global View): The researchers apply Principal Component Analysis (PCA) to the residuals. This step identifies latent "factors" that drive the dynamics of the entire market, such as national economic shifts or broad consumer trends that affect all units simultaneously.
  3. Idiosyncratic Correction (The Local View): This is the framework's core innovation. Most models treat idiosyncratic components—the remaining "noise" after factors are removed—as pure error. FarmTreat argues that these residuals contain valuable information about specific "peer-to-peer" relationships. Using LASSO regression, the model identifies a sparse set of peers whose local fluctuations help predict the treated unit’s behavior.
  4. Counterfactual Projection: The combined model (global factors plus local idiosyncratic corrections) is projected into the post-intervention period. The treatment effect is then estimated as the difference between the actual observed outcomes and this projected counterfactual.

Empirical Evaluation and Data

The authors validated FarmTreat using both simulated data and a novel retail dataset from a major retailer in Brazil. The retail data focused on five products in the "Sweets and Candies" category across more than 1,400 stores in 400+ municipalities during 2016 and 2017. To ensure a balanced experimental design, the researchers incorporated municipal-level covariates such as GDP per capita, population demographics, and the Human Development Index (HDI).

In simulations, FarmTreat consistently outperformed "pure" factor models, which ignore idiosyncratic correlations, and "pure" LASSO models, which often struggle with the dense correlations inherent in raw economic data. The authors also provided theoretical proof that the estimator for the instantaneous treatment effect is unbiased, allowing for valid statistical inference even with small post-intervention windows.

Key Insights and Results

The application of FarmTreat to the Brazilian retail data yielded several significant findings:

  • Statistical Significance: Price changes resulted in statistically significant effects in over 20% of the municipalities in the treatment group, highlighting the high degree of local variation in consumer behavior.
  • Optimal Pricing: The model revealed that for the "Sweets and Candies" category, optimal prices for profit maximization were generally lower than the actual prices set by the company. This suggests that the retailer was overpricing these goods relative to their local price elasticities.
  • The Value of "Noise": By accounting for idiosyncratic residuals, the model significantly reduced the Mean Squared Error (MSE) of the counterfactual estimates. This confirms that what is often discarded as noise in other models is actually a rich source of predictive information.
  • Extreme Heterogeneity: The results underscored that price elasticities vary wildly between locations. A centralized, "one-size-fits-all" pricing strategy is likely sub-optimal compared to localized pricing informed by factor-adjusted models.

Limitations and Future Directions

While FarmTreat represents a significant advancement, the authors noted several areas for improvement. The model currently assumes "Intervention Independence," meaning it does not account for spillover effects where a price change in one city might influence sales in a neighboring city (cross-shopping). Additionally, the model struggled with supply chain disruptions; in cases where a product went out of stock, the model incorrectly attributed the drop in sales to price elasticity rather than lack of availability.

Future research may extend the FarmTreat framework to multivariate cases involving multiple simultaneous treatments or incorporate more complex machine learning architectures to further refine the idiosyncratic correction step. Nevertheless, FarmTreat provides a powerful new tool for researchers seeking to exploit all available information in high-dimensional counterfactual analysis.

Summary updated

Abstract

Optimal pricing, that is determining the price level that maximizes profit or revenue of a given product, is a vital task for the retail industry. To select such a quantity, one needs first to estimate the price elasticity from the product demand. Regression methods usually fail to recover such elasticities due to confounding effects and price endogeneity. Therefore, randomized experiments are typically required. However, elasticities can be highly heterogeneous depending on the location of stores, for example. As the randomization frequently occurs at the municipal level, standard difference-in-differences methods may also fail. Possible solutions are based on methodologies to measure the effects of treatments on a single (or just a few) treated unit(s) based on counterfactuals constructed from artificial controls. For example, for each city in the treatment group, a counterfactual may be constructed from the untreated locations. In this article, we apply a novel high-dimensional statistical method to measure the effects of price changes on daily sales from a major retailer in Brazil. The proposed methodology combines principal components (factors) and sparse regressions, resulting in a method called Factor-Adjusted Regularized Method for Treatment evaluation (FarmTreat). The data consist of daily sales and prices of five different products over more than 400 municipalities. The products considered belong to the sweet and candies category and experiments have been conducted over the years of 2016 and 2017. Our results confirm the hypothesis of a high degree of heterogeneity yielding very different pricing strategies over distinct municipalities. Supplementary materials for this article are available online.
BibTeX
@misc{fan2020do,
  author = {Jianqing Fan and Ricardo Masini and Marcelo C. Medeiros},
  title = {{Do We Exploit all Information for Counterfactual Analysis? Benefits of Factor Models and Idiosyncratic Correction}},
  year = {2020},
  eprint = {2011.03996},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.03996},
}
Figure