Orthogonal Machine Learning for Demand Estimation: High Dimensional Causal Inference in Dynamic Panels

Victor Chernozhukov, Matt Goldman, Vira Semenova, Matt TaddyarXiv 2017 · arXiv

Abstract

There has been growing interest in how economists can import machine learning tools designed for prediction to facilitate, optimize, and automate the model selection process while retaining desirable inference properties for causal parameters. Focusing on partially linear models, this paper extends the Double Machine Learning (DML) framework to allow for (1) a number of treatments that may grow with the sample size and (2) the analysis of panel data under sequentially exogenous errors. In the low-dimensional treatment (LD) regime, the authors show that second-stage ordinary least squares (OLS) estimators attain root-n convergence and desired coverage even if the dimensionality of treatment grows at a rate of $O(N/ \log N)$. In the high-dimensional sparse (HDS) regime, second-stage orthogonal LASSO and debiased orthogonal LASSO exhibit asymptotic properties equivalent to oracle estimators. These advances offer a robust alternative for practitioners estimating short-term demand elasticities in complex, non-contractual retail settings.

Introduction

In modern retail, estimating demand elasticities—the measure of how sales respond to price changes—is complicated by the sheer volume of potential confounding variables. While standard machine learning (ML) models excel at predicting sales based on historical data, they often fail at causal inference. A model might correctly predict high sales during a holiday when prices are also high, but it cannot inherently distinguish whether the sales were driven by the holiday or the price level. To isolate the causal effect of a price change, economists must account for "confounders"—variables that influence both price and sales.

Traditionally, this required manual variable selection, which is subjective and labor-intensive. Furthermore, as the number of potential controls (e.g., product hierarchies, competitor prices, and textual descriptions) grows, standard statistical tools suffer from the "curse of dimensionality." This paper bridges the gap between predictive ML and causal econometrics by extending the Double Machine Learning framework to dynamic panel data.

The Double Machine Learning Framework

The core of this research builds on the Double Machine Learning (DML) framework, which utilizes a two-stage "residualization" process to eliminate bias:

  1. First Stage: ML algorithms (such as Random Forests or Lasso) are used to predict the treatment (price) and the outcome (sales) based on the set of control variables. By subtracting these predictions from the actual values, researchers obtain "residuals." These residuals represent the variation in price and sales that cannot be explained by the observable environment.
  2. Second Stage: The sales residuals are regressed on the price residuals. Because the influence of the controls has been "partialled out," this stage identifies the true causal effect (elasticity).

The "orthogonal" nature of this approach ensures that the second-stage estimator is relatively insensitive to small estimation errors made by the ML models in the first stage, allowing for valid confidence intervals and hypothesis testing.

Extensions for Dynamic Panels and High Dimensionality

The paper introduces several critical innovations to make DML applicable to real-world industrial demand systems:

Sequential Exogeneity

In dynamic retail environments, past sales often influence future pricing decisions. The authors adapt the DML framework to handle "sequential exogeneity," ensuring that the model remains valid even when the data tracks many items over time and incorporates feedback loops between past performance and current strategy.

Growing Treatment Dimensions

In large-scale retail, a researcher might want to estimate unique elasticities for thousands of different products simultaneously. The authors prove that the model remains statistically reliable even if the number of treatments ($d$) grows along with the sample size ($N$), specifically at a rate of $O(N/ \log N)$.

High-Dimensional Sparse (HDS) Regime

When the number of treatments actually exceeds the number of observations ($d > N$), the paper introduces Orthogonal LASSO and Debiased Orthogonal LASSO. These methods leverage the assumption of "sparsity"—the idea that while many factors could influence demand, only a relatively small subset has a significant impact. The authors demonstrate that these estimators are asymptotically equivalent to an "Oracle" estimator that perfectly knows the first-stage functions.

Computational Efficiency via Affine Structure

Estimating cross-price effects for thousands of products is computationally expensive. The authors exploit an "Affine Structure," allowing practitioners to train a single first-stage model for price rather than separate models for every interaction term. This drastically reduces the computational overhead for large-scale demand systems.

Key Insights for Practitioners

  • The Universe of Signals: For a firm setting its own prices, the data used to make those decisions constitutes the "universe" of signals. If the firm can model its own pricing rule using ML, any remaining variation in price acts as a "natural experiment" to identify causal effects.
  • Hierarchical Sparsity: Product elasticities often follow hierarchies (e.g., products within the same category react similarly). This justifies the use of LASSO in the second stage to identify specific products with unique sensitivities.
  • Model Robustness: By partialling out controls, the second-stage results become robust to the specific choice of ML algorithm used in the first stage, provided the algorithm has sufficient predictive power.

Conclusion and Future Work

This work provides a rigorous mathematical foundation for using high-dimensional ML in causal demand estimation. While it offers a powerful tool for automated model selection, the authors note that human expertise is still required to avoid "bad controls"—variables that are themselves affected by the price decision. Future research may focus on proving these rates for arbitrary cluster sizes in unobserved heterogeneity and conducting large-scale empirical comparisons against traditional demand models like the Berry-Levinsohn-Pakes (BLP) framework.

Summary updated

Abstract

There has been growing interest in how economists can import machine learning tools designed for prediction to facilitate, optimize and automate the model selection process, while still retaining desirable inference properties for causal parameters. Focusing on partially linear models, we extend the Double ML framework to allow for (1) a number of treatments that may grow with the sample size and (2) the analysis of panel data under sequentially exogenous errors. Our low-dimensional treatment (LD) regime directly extends the work in Chernozhukov et al. (2016), by showing that the coefficients from a second stage, ordinary least squares estimator attain root-n convergence and desired coverage even if the dimensionality of treatment is allowed to grow at a rate of O(N/ log N ). Additionally we consider a high-dimensional sparse (HDS) regime in which we show that second stage orthogonal LASSO and debiased orthogonal LASSO have asymptotic properties equivalent to oracle estimators with known first stage estimators. We argue that these advances make Double ML methods a desirable alternative for practitioners estimating short-term demand elasticities in non-contractual settings.
BibTeX
@misc{chernozhukov2017orthogonal,
  author = {Victor Chernozhukov and Matt Goldman and Vira Semenova and Matt Taddy},
  title = {{Orthogonal Machine Learning for Demand Estimation: High Dimensional Causal Inference in Dynamic Panels}},
  year = {2017},
  eprint = {1712.09988},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1712.09988},
}
Figure