Comment on "Sequential validation of treatment heterogeneity" and "Comment on generic machine learning inference on heterogeneous treatment effects in randomized experiments"

Victor Chernozhukov, Mert Demirer, Esther Duflo, Iván Fernández‐ValarXiv 2025 · DOI · arXiv

This paper is a formal reply by Victor Chernozhukov and colleagues to two major critiques of their work on using machine learning to understand Heterogeneous Treatment Effects (HTE). In the world of randomized controlled trials (like medical or policy experiments), we don't just want to know if a treatment works on average; we want to know who it works for. This paper defends the "multiple-split" approach as a robust way to handle the statistical noise and "data mining" risks inherent in modern ML-based inference.

TL;DR

The authors argue that while single-split data methods are computationally faster, "multiple-split" approaches (running an analysis many times with different random data partitions) significantly reduce estimation risk and protect against "p-hacking." By combining their framework with new sequential inference techniques, they show it's possible to achieve higher statistical power while keeping false discovery rates near zero.

What Problem Does This Paper Address?

When using machine learning to estimate the Conditional Average Treatment Effect (CATE)—the predicted effect of a treatment for a specific individual based on their characteristics—researchers usually split their data into a "training" set (to find patterns) and a "testing" set (to validate them).

However, this creates two major headaches:

  1. Splitting Noise: Your results might change drastically depending on which specific rows of data ended up in your training vs. testing sets.
  2. Data Mining: A researcher could (intentionally or not) try several different random seeds for their data split and only report the one that shows a "significant" result.

The authors address whether their proposed "multiple-split" method (CDDF) is better than newer, single-split alternatives proposed by Stefan Wager (2024) and Imai & Li (2024).

Methods

The paper compares three main statistical philosophies:

  • The CDDF Approach (Multiple-Split): The authors' original method. It involves splitting the data many times (e.g., 100 times), calculating the treatment effects for each, and then taking the median. This "smooths out" the luck of the draw from any single split.
  • Sequential Inference (Wager, 2024): A method that uses "martingale" properties (a concept from probability theory) to process data folds one after another, allowing for a single-split p-value that is more powerful than a simple median.
  • Analytical Single-Split (Imai & Li, 2024): A method (IMLI) based on Neyman’s randomization paradigm that aims to account for splitting uncertainty mathematically without needing to run the model 100 times.

The authors also introduce a "Mining" simulation, where they emulate a researcher who tries five different random seeds and picks the one with the most "favorable" (positive) result to see which method breaks first.

Datasets

The paper relies on Monte Carlo simulations rather than a specific empirical dataset.

  • Sample Size: 1,000 observations.
  • Replications: 10,000 simulation runs to ensure results are statistically sound.
  • Scenarios: They test a "Null" scenario (where the treatment does absolutely nothing) to check for false positives, and a "Signal" scenario (where the treatment effect depends on a specific variable $z_1$) to check for statistical power.

Key Insights and Ideas

  • Stability is a Shield: The "aha moment" here is that multiple splitting isn't just about getting a "better" average; it's a structural defense against p-hacking. Because the median of 100 splits is much harder to manipulate than a single split, it naturally discourages "seed searching."
  • Synergy, not Competition: The authors found that Wager’s sequential approach and their multiple-split approach aren't mutually exclusive. When combined, they produce a "strong winner" that has the highest power to find real effects while keeping false rejections extremely low.
  • The Cost of Speed: While single-split methods (like IMLI) are computationally "cheaper," they carry higher Estimation Risk. The authors show their method has a lower Mean Absolute Deviation (MAD), meaning their estimates are consistently closer to the truth.

Key Results

  • False Rejections: In the "Naïve" single-split setup, the false rejection rate was 8.40% (higher than the intended 5%). Moving to "Multiple-Splits" crashed that error rate down to 0.81%.
  • Power: Combining sequential inference with multiple splits achieved a power of 64.68%, significantly outperforming the standard 2-fold split (44.73%).
  • Resistance to Mining: When simulating a "mining" researcher (trying 5 seeds), the single-split IMLI method saw its false rejection rate jump to 1.66%, while the multiple-split CDDF method remained rock-solid at 0.01%.

Limitations and Future Work

The authors acknowledge that their method is computationally demanding. Running an ML model 100 times is fine for a dataset of 1,000 people, but it might be a bottleneck for massive industrial datasets.

They also note that while their method excels when the CATE function is "special" (e.g., zero, flat, or sparse), more research is needed to balance the trade-offs between computational efficiency and robustness in more complex, non-sparse environments. The authors conclude by welcoming the alternative methods as "useful complements" rather than replacements.

Summary updated

Abstract

We warmly thank Kosuke Imai, Michael Lingzhi Li, and Stefan Wager for their gracious and insightful comments. We are particularly encouraged that both pieces recognize the importance of the research agenda the lecture laid out, which we see as critical for applied researchers. It is also great to see that both underscore the potential of the basic approach we propose - targeting summary features of the CATE after proxy estimation with sample splitting. We are also happy that both papers push us (and the reader) to continue thinking about the inference problem associated with sample splitting. We recognize that our current paper is only scratching the surface of this interesting agenda. Our proposal is certainly not the only option, and it is exciting that both papers provide and assess alternatives. Hopefully, this will generate even more work in this area.
BibTeX
@misc{chernozhukov2025comment,
  author = {Victor Chernozhukov and Mert Demirer and Esther Duflo and Iván Fernández‐Val},
  title = {{Comment on "Sequential validation of treatment heterogeneity" and "Comment on generic machine learning inference on heterogeneous treatment effects in randomized experiments"}},
  year = {2025},
  eprint = {2502.01548},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.01548},
  howpublished = {arXiv},
}
Figure