Real Outcomes, Simulated Compliance: An Honest Test Bed for Machine-Learning LATE Estimators

When people are randomly offered a treatment but choose for themselves whether to take it, the standard causal quantity is the local average treatment effect (LATE): the effect on "compliers," the people who take the treatment only because they were offered it. Machine-learning versions of this estimator now exist. Deep neural instrumental-variable (IV) models, doubly and multiply robust learners, and distributional variants all claim to recover it more accurately, and to recover how it varies across people. Almost none of those claims have been checked on real outcomes. Every reported comparison uses outcome functions the authors wrote themselves. The treatment-effect literature already shows why this is dangerous: on a standard semi-synthetic benchmark, neural networks beat forests only because of how the simulated outcome was written. Making the effect additive reversed the ranking (Really Doing Great at Estimating CATE?).

This note proposes a way to build IV benchmarks whose outcomes are real and whose ground truth is still known. We start from a real randomized trial with real outcomes. We then simulate only who complies, by rejection-sampling units so that a synthetic offer, a synthetic compliance type, and the trial's real treatment line up. Because the simulated compliance types depend only on baseline covariates, the true complier effect can be computed from the full original trial by a simple difference in means. The outcome surface, which is the part that biased earlier benchmarks, is never authored. Our central empirical question is whether the reported superiority of neural IV estimators survives when the outcome function comes from the world rather than from the paper.

Core Claim

A randomized trial with outcome $Y$ and randomized binary arm $W$ can be turned into an imperfect-compliance IV dataset with a known answer. Define "treatment" as $W$ itself. For a two-arm trial, assignment is complied with by definition, so the treatment $W$ is fully complied. Then:

  1. Assign each unit a latent compliance type $T \in \{\text{complier}, \text{always-taker}, \text{never-taker}, \text{defier}\}$ using a chosen function of baseline covariates. Some of those covariates will be hidden from the estimators.
  2. Draw a fresh randomized offer $Z$ for each unit.
  3. Compute the take-up $D$ that type $T$ implies under offer $Z$.
  4. Keep the unit only if $D = W$, so that its observed real outcome is the correct potential outcome.

The retention rule works as follows. With original assignment probability $p = P(W=1)$, a unit is retained with probability proportional to

$$ a(T, Z) = \frac{1}{P(W = D(T,Z))}, $$

normalized by its maximum. Here $D(T,Z)$ is the take-up implied by type $T$ under offer $Z$. This makes every (type, offer) combination equally likely to survive. As a result, $Z$ in the retained sample is independent of type and of both potential outcomes. For a 1:1 trial the rule reduces to keeping half the units at random. For JTPA's unequal allocation, the weights correct for it.

The true complier effect is then

$$ \tau_C = E[Y \mid W=1, T=\text{complier}] - E[Y \mid W=0, T=\text{complier}], $$

computed on the full base trial. It is unbiased because $W$ was randomized and $T$ is a pre-treatment function. The same holds for any subgroup of compliers, which gives real ground truth for conditional LATE.

The construction has four useful properties:

  • Hidden confounding. When the type depends on covariates that also predict the real outcome but are withheld from the estimator, take-up is confounded with the outcome exactly as in practice.
  • Controllable violations. Defier fractions, complier share (instrument strength), and the shape of the compliance mechanism are experimenter-controlled dials.
  • Built-in correctness check. The simple Wald ratio from Identification and Estimation of Local Average Treatment Effects must be unbiased for $\tau_C$ whenever there are no defiers. Its behavior therefore tests that the construction is correct.
  • Precedent. To our knowledge from the indexed material, no IV/LATE paper builds benchmarks this way.

Research Program

Base trials. We will use three, chosen from the file.

  • Twins (heavier versus lighter twin, one-year mortality). Both potential outcomes are observed per pair, so it gives unit-level ground truth. We treat the heavier-twin indicator as $W$ assigned at random within pair.
  • Oregon Health Insurance Experiment (OHIE). We use the lottery win as $W$ and utilization and financial outcomes as $Y$. Because the lottery was stratified by household size, we condition on household size.
  • JTPA. We use randomized assignment as $W$ and earnings as $Y$, with 2:1 allocation handled by the acceptance weights.

In OHIE and JTPA the "treatment" becomes the offer-level effect of the original trial. That is a real causal effect with a real, messy covariate dependence, which is what we need.

Compliance mechanisms. These are authored, which is the one place the construction could re-introduce bias. We therefore draw them from deliberately contrasting families:

We sweep three dials:

  • complier share, from strong instruments down to weak ones where the denominator of the Wald ratio approaches zero;
  • defier fraction, from zero to 0.2, motivated by the defier traces that Poulos reports in OHIE;
  • a transparent exclusion leak, $Y \leftarrow Y + \delta Z$ for a small $\delta$. This is the one explicitly synthetic outcome perturbation, and we label it as such.

What gets released. We will release the generator, fixed seeds, the ground-truth tables with their own sampling uncertainty, and an evaluation harness. Following the heavy-tail finding of Really Doing Great at Estimating CATE?, the harness reports medians and upper quantiles of error across resamples, not only means.

Baselines

Naive references. The first is intention-to-treat, the offer-versus-no-offer difference. We report it to show how badly it is diluted relative to $\tau_C$, not as a LATE estimator. The second is the as-treated comparison of takers against non-takers. The hidden-covariate compliance mechanisms should make it visibly biased, mirroring the confounding failure of plain feed-forward networks in Counterfactual Prediction with Deep Instrumental Variables Networks.

Classical references. The Wald ratio and covariate-adjusted linear two-stage least squares (2SLS) are the "do nothing clever" estimators. Under our construction the Wald ratio is unbiased for the pooled LATE whenever defiers are absent. Any machine-learning method must therefore beat it on variance or on heterogeneity, not on bias.

Machine-learning estimators, the main competitors.

  • MRIV-Net (Estimating individual treatment effects under unobserved confounding using binary instruments) is a multiply robust pseudo-outcome learner for binary instruments. It is the strongest published method for conditional complier effects in exactly this binary setting, and it was reported to beat DeepIV, DMLIV and Kernel IV on semi-synthetic OHIE.
  • DFIV (Learning Deep Features in Instrumental Variable Regression) learns neural features with closed-form ridge stages.
  • The orthogonal distributional estimator of Beyond the Average is used for distributional complier effects on OHIE's count outcomes.
  • DMLIV is included as a doubly robust reference.
  • A forest-based IV estimator provides the tree-inductive-bias contrast that the CATE critique says matters. No paper in the file supplies one, so we implement it from standard components.
  • DeepIV is included for completeness. It is expected to struggle, given DFIV's report that its density stage fails in high dimensions.

Reported numbers. We found no reported scalar result on Lacuna for MRIV-Net, DFIV, or the distributional estimator on any benchmark we use. The file gives only qualitative rankings and figures. The only scalar IV result in the file is DWLS's JTPA root mean squared error of 252 ± 141 versus 946 ± 663 for minimax least squares, from Estimation of Local Average Treatment Effect by Data Combination. That comes from an unlinked-data design and is not directly comparable to ours. We will quote it only as context, and we will add DWLS as a baseline on the JTPA base trial when run in its two-regime form.

Fairness. Every method gets the same retained samples and the same observed covariates. Each gets an equal hyperparameter budget, chosen by a criterion computable without ground truth: held-out first-stage and reduced-form loss. We also report an oracle-tuned run to separate method quality from tuning failure. DFIV's authors flag tuning as unsolved, so this split matters.

What counts as winning. A method "wins" a cell only if its median error is below the Wald or 2SLS reference by more than the ground-truth noise floor. We will carry this out consistently across compliance families.

Ablations. The key ingredient of the proposal is that outcomes are real. The central ablation therefore replaces the real $Y$ with fitted synthetic outcome surfaces of two kinds, one smooth and non-separable and one additive, while holding everything else in the construction fixed. That isolates exactly what real outcomes change.

Why not just…?

Why not just add more synthetic outcome functions, some tree-friendly and some neural-friendly, as the CATE critique did? This is the most serious alternative. The reversal in Really Doing Great at Estimating CATE? was found this way. But a synthetic sweep only shows that rankings can flip. It cannot say which ranking holds on real data, because every synthetic family is authored. Our ablation settles this directly. If rankings on one synthetic family predict the real-outcome rankings as well as the real outcomes predict themselves across resamples, the reviewer is right and cheap synthetic sweeps suffice. We will report that as a finding.

Why not just use the Wald ratio and stop? Under our construction it is unbiased for pooled LATE, so on that target it may well tie or win. That would itself be informative given how many papers claim otherwise. Machine-learning methods can only justify themselves on variance under covariate adjustment, on subgroup complier effects, on distributional effects, and on robustness to defiers and weak instruments. Those are the metrics we emphasize. We will say plainly where Wald is not beaten.

Why not just validate against published estimates on real noncompliance trials like OHIE? Those trials have no known complier effect. Agreement between an ML estimator and the Wald estimate on OHIE shows consistency, not correctness, and both share the same untestable assumptions. Our construction knows the defier fraction and the exclusion leak by design. Real trials cannot provide that.

Why not just select models by held-out predictive fit? Good first-stage or reduced-form fit need not imply good complier-effect estimates. Whether it does is an open question, flagged by the DFIV authors. We measure it directly: we report the rank correlation between each tuning criterion and true error.

Proposed Experiments and Decisive Checks

Sanity check. Without defiers, the Wald ratio's error must be centered at zero within the noise floor on all three base trials. If it is not, the construction is broken.

Headline test. We compute the Kendall rank correlation between estimator rankings on existing-style synthetic IV setups and rankings on real-outcome datasets, for the same covariates and compliance mechanisms. A low or negative correlation would extend the CATE benchmarking critique to IV methods with evidence rather than conjecture.

Degradation curves. We plot estimator error and confidence-interval coverage against three quantities: complier share, defier fraction, and exclusion leak. None of the ML-IV papers in the file report these curves.

Decision metrics. On Twins, where unit-level truth exists, we compare conditional-effect error with the policy regret of targeting encouragement to predicted high-effect compliers. This tests whether ranking by root mean squared error agrees with ranking by decision value, the gap the CATE critique raised but did not examine for IV.

Stability across compliance mechanisms. We check whether the real-outcome ranking holds across all four compliance families. If it does not, the remaining source of benchmark bias is the compliance mechanism, and we will state which family drives it.