Cheap, Causally Honest Architecture Search for Treatment-Effect Networks

Neural networks that estimate individual treatment effects (how much a specific patient would gain from a drug compared with no drug) are built by hand. Typical designs are a shared trunk with separate output heads for treated and untreated outcomes, sometimes with an extra head that predicts who gets treated. Their depth, width, branching point and regularization weights are hard to tune, because the quantity we care about, the error on the unobserved counterfactual outcome, cannot be measured on a validation set. The existing remedy is to train every candidate fully. One then ranks the candidates with a "doubly robust" score: a validation criterion built from separately fitted models of treatment assignment and of outcomes, which stays approximately unbiased if either of those models is right. That works, but it is expensive. Cheap architecture-search tools, such as scoring networks at initialization or after a few epochs, all rank candidates by ordinary predictive loss. Causal inference has shown that ordinary predictive loss is the wrong target.

This note proposes to make causal architecture search cheap without making it causally blind. Three choices do this. The doubly robust validation targets are computed once, with sample splitting, and then frozen. Scoring any candidate at any checkpoint then costs a single forward pass. Early-stopping-based search (successive halving: train many candidates briefly, keep the best fraction, train those longer) promotes candidates by that frozen causal score instead of factual loss. And we build a small exhaustive benchmark of fully trained treatment-effect architectures with per-epoch logs. With it we can measure directly when, during training, the causal ranking of architectures becomes trustworthy. The headline question is whether low-budget causal scores keep the ranking of the best architectures. The answer bears on more than cost: the treatment-effect literature has shown that factual loss rankings can be confidently wrong.

Core Claim

Our hypothesis is about the top of the ranking. With early-checkpoint rankings, factual validation loss will have high overall correlation with final factual loss but poor correlation with true counterfactual error. A frozen doubly robust score will be noisier early in training but will rank the top tier of architectures correctly at a fraction of full training. The result would be a search that matches full-training model selection on true error at roughly an order of magnitude less compute.

The motivating evidence is from Adapting Neural Networks for the Estimation of Treatment Effects. It introduced Dragonnet, a three-headed network with propensity, treated-outcome and control-outcome heads, plus a "targeted regularization" term from semiparametric theory. It found that models with worse factual prediction can give better effect estimates. It also found that training the propensity head jointly with the outcome heads beats a frozen two-stage version.

The selection metric comes from Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation. It benchmarked 34 surrogate metrics over 415 fixed estimators on 78 datasets and found doubly robust and TMLE-based T-scores globally dominant when their nuisance models are tuned with AutoML. That study selected among fully trained, static configurations. It never asked whether the same scores keep their ranking power under partial training. That question is exactly what makes the scores usable inside a search loop.

The search mechanics come from three papers, all of which rank by factual loss on standard supervised tasks:

We replace their ranking signal and keep their scheduling.

Research Program

Nuisance models. For each dataset we split the training data into folds. On each fold we fit a propensity model $\hat e(X)$ and outcome models $\hat\mu_1(X), \hat\mu_0(X)$ out of fold, tuned with AutoML in the manner of the model-selection study above. From these we compute one doubly robust pseudo-outcome per unit:

$$\tilde Y_i = \hat\mu_1(X_i) - \hat\mu_0(X_i) + \frac{T_i\,(Y_i - \hat\mu_1(X_i))}{\hat e(X_i)} - \frac{(1-T_i)\,(Y_i - \hat\mu_0(X_i))}{1 - \hat e(X_i)}.$$

Here $X_i$ are covariates, $T_i$ the binary treatment and $Y_i$ the observed outcome. A candidate network $f$ with effect prediction $\hat\tau_f$ is scored by

$$S(f) = \frac{1}{n}\sum_i \big(\tilde Y_i - \hat\tau_f(X_i)\big)^2.$$

These nuisance models are deliberately separate from the candidate's own propensity head, so that a candidate is never graded by its own nuisance estimates. Because $\tilde Y$ is fixed, scoring a candidate at any checkpoint is a forward pass. The cost of causal evaluation thus drops from "refit nuisances per candidate" to essentially zero. Inverse weights are clipped at a pre-registered threshold, and we report overlap diagnostics, because extreme propensities are a known failure mode of these weights.

Leakage control. The pseudo-outcomes are split into a promotion half and a held-out selection half. Successive-halving decisions use only the promotion half. Final selection uses the held-out half. True counterfactual error, available only on semi-synthetic benchmarks, is used solely for reporting. This addresses the leakage trap noted in Can Training Logs Make Model Comparisons More Precise?, where reusing a metric that already drove selection invalidates later inference.

Search space. The space is the Dragonnet family:

Over-weighting that balance penalty is known to strip out predictive signal. Its weight is therefore exactly the kind of choice factual loss will misjudge.

Exhaustive benchmark. On semi-synthetic datasets we train a grid of on the order of a thousand configurations to completion, with several seeds each. These are small tabular networks, so this is affordable once. At every epoch we log factual loss, propensity loss, balance discrepancy, the promotion-half doubly robust score, and the true counterfactual error. This plays the role NAS-Bench-201 plays for vision. It turns every search-strategy comparison into a table lookup. It also lets us draw the central object of study: the rank correlation between each early-epoch signal and final true error, as a function of training fraction, separately for the whole space and for the top 10%.

Baselines

Default architecture. The do-nothing reference is Dragonnet with targeted regularization at its published default architecture, trained with the same seeds. The strongest reported numbers for this design are an ATE mean absolute error of 0.35 on ACIC 2018 (versus 1.45 for TARNET) and an average-effect error $\Delta_{\text{all}} = 0.11$ on IHDP, both from Adapting Neural Networks for the Estimation of Treatment Effects. Our primary metric is root counterfactual error (root PEHE) of the selected network, which those numbers do not cover. So we re-run the default on identical splits and report both quantities. Beating it means a statistically lower root PEHE across dataset replications. If the default sits within seed noise of the best configuration in the space, that is a finding against the value of search here.

Full-training selection. The strongest existing method is to train every candidate in the space fully and select by the AutoML-tuned doubly robust T-score, as recommended by Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation. That paper reports dominance qualitatively and as a share of wins. We found no single reported regret number on Lacuna for this procedure on IHDP or ACIC, so we measure it ourselves on the exhaustive benchmark. It gets the full compute budget and the same nuisance models. Our method "beats" it if it reaches equal or lower root-PEHE regret (gap to the best architecture in the space) at no more than a tenth of its candidate-epochs. Matching it at lower cost is the main target; we do not expect to exceed it in accuracy.

Relative-error selection. A second strong selector is the pairwise relative-error test from A Relative Error-Based Evaluation Framework of Heterogeneous Treatment Effect Estimators, which reports 80% to 94% selection accuracy on IHDP and Twins. Its cost grows with the square of the number of candidates. We therefore apply it only to the final shortlist, as a possible replacement for the held-out score in the last step, at matched compute.

Key ablation. The ablation that isolates our ingredient is identical successive halving with identical budgets and rungs, promoting by factual validation loss instead of the doubly robust score. A further ablation uses pTProxy from pTNAS: Progressive Neural Architecture Search for Tabular Data as a zero-cost first filter. All methods share data splits, seeds, search space and total candidate-epoch budget.

Why not just…?

Why not just tune Dragonnet's hyperparameters with ordinary random search and select by the doubly robust score? That is the full-training baseline above, and it is the honest gold standard. It is not wrong, only expensive: every candidate pays full training, and on small datasets several seeds. Our claim is narrowly that most of that cost buys nothing. The exhaustive benchmark shows this directly if top-tier doubly robust rankings stabilize early. If they do not stabilize until late in training, random search with full training is the right answer, and we will say so.

Why not just run successive halving on factual validation loss, as every existing budgeted search does? This is the most serious threat. The prediction–estimation trade-off reported for Dragonnet suggests factual loss misranks causal quality, but we have no direct evidence about early-epoch rankings. On datasets with strong shared structure between potential outcomes, factual loss and causal error may well co-vary. Really Doing Great at Estimating CATE? showed benchmark data-generating processes strongly shape which estimator family wins, so this may differ across datasets. The matched ablation settles it per dataset. If factual-loss halving matches ours, the doubly robust promotion step is unnecessary, and the contribution reduces to the benchmark and the fidelity curves.

Why not just score candidates at initialization with a zero-cost proxy? Existing proxies measure trainability and expressivity for factual prediction. Even on plain tabular data, vision-derived proxies fall below a Spearman correlation of 0.70 against pTProxy's 0.82, per pTNAS: Progressive Neural Architecture Search for Tabular Data. Whether an untrained network controls confounding well depends on what it learns about treatment assignment, which is not present at initialization. We test pTProxy as a coarse filter. We expect it to be useful only for discarding clearly undersized networks.

Why not just use a causal forest and skip neural search? Forests do well when effects are additive, according to the critical benchmarking study, while representation networks do better under nonlinear confounding. We include a causal forest as an external reference on every dataset. Any dataset where it beats the best network in the space is reported as one where neural search is moot.

Proposed Experiments

Datasets. We use IHDP (747 units, many outcome replications), the 77 ACIC 2016 data-generating processes, ACIC 2018 and Twins. All provide ground-truth effects for evaluation.

Fidelity analysis. The first experiment is the fidelity analysis on the exhaustive benchmark. For training fractions from 1/27 to full, we compute Kendall tau between each early signal and final root PEHE, over the whole space and over the top 10%. The signals are factual loss, the doubly robust score, propensity loss, balance discrepancy and pTProxy. The decisive check is whether the doubly robust score's top-tier correlation exceeds factual loss's at budgets of one-ninth or less. If it does not, the core claim fails.

Search comparison. The second experiment runs each search strategy at matched candidate-epoch budgets on the benchmark lookups and reports root-PEHE regret versus compute. HyperTLearner's out-of-sample PEHE of 1.12 on IHDP, from Dynamic Inter-treatment Information Sharing for Individualized Treatment Effects Estimation, serves as an external sanity anchor for whether the searched networks are competitive at all.

Stress tests. The third experiment stresses the method where it should break. On ACIC configurations with poor overlap, it measures how weight clipping and pseudo-outcome noise degrade promotion decisions. We also vary the number of seeds per candidate down to one to find the budget below which noise in the causal ranking swamps the gain. That is the causal counterpart of the low-run failure documented for training-log adjustments.