Site-Held-Out Credit for Searching Treatment-Effect Feature Sets

Many treatment-effect models fail in a simple way. A model is supposed to predict who benefits from an intervention more than others, but on real trial data it usually does worse than predicting the same average effect for everyone. One large benchmark found that 62% of fitted models were worse than predicting zero effect for everyone, and 80% of the rest failed to beat a constant average-effect model (Do Contemporary CATE Models Capture Real-World Heterogeneity?). One plausible culprit is the covariate set. Clinical and survey data carry dozens to hundreds of overlapping variables. Some of them only drive who received treatment, and including those provably inflates variance (High-Dimensional Feature Selection for Sample Efficient Treatment Effect Estimation). Others encode site-specific habits that do not carry over to a new hospital or country. We propose to search over which covariates the effect model is allowed to use, and to score candidate covariate sets by how well they beat the constant-effect baseline on a held-out site.

The search uses adaptive operator selection, meaning a small bandit learner that decides online which edit move to try next (for example "add a variable that predicts the outcome" or "drop a variable that mainly predicts treatment"), based on which moves have recently paid off. The central idea is the scoring signal, which we call the credit. Individual treatment effects are never observed, so any search score is a noisy stand-in. A score computed inside a single cohort rewards exactly the redundant, site-specific variables that cause external failure. Computing the score on a held-out site turns external cohort validation from a final report into the training signal for the search.

Core Claim

Searching feature subsets for conditional average treatment effect (CATE) estimation is only useful if three things hold. First, the search credit must be a ground-truth-free improvement over the constant-effect baseline. Second, that improvement must be measured on a site the estimator did not see. Third, the final subset must be confirmed on a separate untouched site under error-controlled selection. CATE here means the average treatment benefit for people with covariates $x$, written $\tau(x)$.

Concretely, suppose we have cohorts $k = 1,\dots,K$. For a candidate subset $S$ we fit a fixed base learner on all search sites except $k$. On site $k$ we then compute a doubly robust pseudo-outcome loss $L_k(S)$, an unbiased-in-expectation proxy for error in $\tau$, of the kind used by Reliable Selection of Heterogeneous Treatment Effect Estimators. We compute the same loss $L_k(\bar\tau)$ for the constant-effect model fitted on the same training sites. The credit is the worst-site gain over that baseline:

$$R(S) = \min_{k} \big[ L_k(\bar\tau) - L_k(S) \big].$$

A positive $R(S)$ means the subset yields heterogeneity that beats the constant effect on every held-out search site. The minimum is chosen rather than the mean so that a subset winning big on one site cannot hide a loss elsewhere. Using $\min_k$ is a design choice we test against the mean.

We make one prediction that is not obvious. Credit computed within a single cohort will systematically favor larger subsets that contain treatment-assignment drivers and site-coded proxies. Site-held-out credit will favor smaller subsets built from outcome predictors, and it will lower the degeneracy rate on the final untouched site.

Research Program

The search operates on binary inclusion masks over covariates, starting from a small seed set. It uses four operator families, each tied to the covariate roles of High-Dimensional Feature Selection for Sample Efficient Treatment Effect Estimation. Those roles are treatment-only drivers $X_1$, confounders $X_2$, and outcome-only predictors $X_3$. The four families are:

  • adding a variable ranked high by a cheap outcome-model importance score;
  • dropping a variable whose propensity importance greatly exceeds its outcome importance (a likely $X_1$);
  • swapping a variable for its most correlated unselected neighbor, which targets redundancy directly;
  • a uniform random flip, kept as an exploration move.

These role estimates come from fast nuisance models and are deliberately rough. The point of the bandit is to learn which move is productive given how wrong those estimates are on a particular dataset.

Operator choice uses dynamic Thompson sampling with a concentration cap, the method of Adaptive Operator Selection Based on Dynamic Thompson Sampling for MOEA/D. We reuse it because the cap lets old evidence decay. Operator usefulness should change during the search: adding outcome predictors pays early, while pruning treatment drivers and redundant copies pays later. Rewards are rank-based improvements in $R(S)$ rather than raw values, following Fitness-AUC bandit adaptive strategy selection vs. the probability matching one within differential evolution. That paper showed rank credit is robust to monotone rescaling, which matters here because the pseudo-outcome loss has heavy tails.

The base learner is held fixed: an S-learner with gradient-boosted trees, the most reliable model in the Yu and Sun benchmark. This isolates feature selection from model choice. After the search, the top few subsets go through the familywise-error-controlled selection of Reliable Selection of Heterogeneous Treatment Effect Estimators on a final site that the search never touched. That paper showed naive selection among noisy candidates produces false winners, and a search that evaluates thousands of subsets is the extreme case of that problem.

Prior work plays three roles here. It motivates the gap: the Yu and Sun degeneracy findings, and the external-collapse literature such as Radiomics in prostate cancer imaging for a personalized treatment approach and Developing and validating COVID-19 adverse outcome risk prediction models. It constrains the design: Guo and Gao on false selection, and Multi-Source Causal Feature Selection, which shows full Markov-blanket discovery breaks at small sample sizes, so we do not attempt it. It supplies methods: dynamic Thompson sampling, rank credit, and doubly robust pseudo-outcomes.

The approach also differs from three related lines of work. Multi-Accurate CATE is Robust to Unknown Covariate Shifts corrects a fixed model's bias using a small external randomized audit set but leaves its inputs and variance untouched, whereas we change the inputs. DRCFS selects direct causes of the outcome, which are not the same as modifiers of the treatment effect. The jointly sparse method uses a static linear penalty, whereas our search is model-agnostic and its score is external.

Baselines

The naive reference points are the constant average-effect model and the zero-effect model. They define degeneracy, and any method must beat them on the untouched site to count as producing heterogeneity at all.

The strongest existing competitor is the S-learner with cross-validated gradient boosting on the full covariate set, as evaluated in Do Contemporary CATE Models Capture Real-World Heterogeneity?. It reports the highest win share, 25.5%, and the lowest degeneracy rate, 6.3%, across its 43,200 dataset variants. Using the same base learner makes it the exact "no selection" counterpart of our method. We run it with the same tuning budget per fit and the same site splits. Beating it means a lower degeneracy rate and a higher win share against the constant-effect model on the untouched site, with paired tests across dataset variants.

The second competitor is that same learner followed by multi-accuracy auditing on a small external slice, per Kern, Kim and Zhou. The file reports no scalar result for this method on Lacuna; its WHI gains are shown only in figures. It is run with the identical audit slice size, and we also test the combination of our subset with auditing.

The third competitor is the jointly sparse MCP/SCAD selector from High-Dimensional Feature Selection for Sample Efficient Treatment Effect Estimation, feeding the same S-learner. Its reported IHDP average-effect estimate is 4.49 against a true 4.36, versus 4.76 without sparsity.

Two simple two-stage filters complete the set. One is an mRMR filter with cardinality tuned by internal cross-validation. The other is a "drop propensity-dominant variables" rule that uses the same role scores as our operators but applies them once, with no search.

Two ablations isolate the ingredients. The first keeps the search and operators but replaces site-held-out credit with within-cohort cross-fitted credit, which tests the core claim. The second keeps the credit but chooses operators uniformly at random, which tests whether the bandit matters. All search variants get the same budget in base-learner fits.

Why not just…?

Why not just run the boosted S-learner on all covariates and audit it? This is the real threat. Trees already ignore many useless variables, and auditing fixes subgroup bias. Our counter-claim is narrower. Auditing does not target variance, as Kern et al. note, and trees still split on site-coded redundant variables when those variables help within a cohort. If audited full-feature S-learning matches our degeneracy rate on untouched sites, the claim fails, and the experiment is designed to show that outcome plainly.

Why not just use a sparse penalty like MCP or Lasso? Its guarantees assume a linear outcome model. DRCFS reports that Lasso-type selection collapses under nonlinear outcomes, and linear double machine learning degenerated in over 90% of cases in some settings of the Yu and Sun benchmark. Even so, it is cheap. If it matches our method on the shifted-site split, we will report that selection helps but search does not.

Why not just apply the propensity-dominance rule once, or search with random operators? This is where adaptive operator selection could prove unnecessary. The uniform-operator ablation and the one-shot rule are run head to head. If either matches the bandit at equal fit budget, we drop the bandit and present the site-held-out credit with a plain search as the contribution. The bandit only earns its place if its learned operator allocation shifts measurably over the search and that shift buys better subsets per fit.

Proposed Experiments

Semi-synthetic controlled study. We use IHDP, ACIC 2016, and the IBM causal benchmark, padded with three kinds of injected covariates: treatment-only drivers, near-duplicate copies of true modifiers, and "site-tag" variables. Site-tag variables correlate with the outcome inside each of several pseudo-sites created by resampling with different selection mechanisms, but not across them. This lets us measure square-root PEHE, the root mean squared error of the estimated effect against true individual effects, which is computable here because the effects are simulated. It also lets us check directly whether the selected subsets exclude the injected harmful variables. For scale reference, Budgeted Heterogeneous Treatment Effect Estimation reports square-root PEHE of 19.6 for its method versus 49.7 for counterfactual regression on IBM at 1,500 queries. That is a different, budgeted regime, so we cite it as context rather than as a target.

Real-trial study. We use the randomized trials in the Yu and Sun benchmark. Where a trial has natural site or country identifiers, we use them as cohorts; the International Stroke Trial is the most promising candidate, conditional on the identifiers being in the release. Otherwise we build pseudo-sites from the benchmark's own observational sampling protocol, with a different selection bias per site. Evaluation uses that benchmark's ground-truth-free $\hat{Q}$ metric, its degeneracy definition, and win share against the constant-effect model, always computed on the untouched site.

Decisive checks. Three results settle the proposal. The first is whether site-held-out credit beats within-cohort credit on untouched-site degeneracy. This is the core claim, and a null result here sinks the proposal. The second is whether the full method beats the audited full-feature S-learner on the 6.3% degeneracy and 25.5% win-share reference. The third is whether the bandit beats uniform operator choice at equal fit budget. We will also report how often the familywise-error-controlled final selection declares no subset better than the constant effect. An honest "no heterogeneity found" answer is a valid output of the method, not a failure of it.