Cheap-First Feature Elimination for Treatment Effect Models

Suppose you want to learn who benefits from a treatment, such as which students respond to a financial-aid reminder or which patients respond to a therapy, from data with hundreds or thousands of recorded attributes. To keep the model accurate and interpretable, you want to throw away the attributes that do not matter. The standard tool is recursive feature elimination (RFE): train a model, drop the least useful features, retrain, and repeat. Two things make this hard for treatment effect models, which estimate how each individual's outcome would change under treatment. Retraining a causal neural network or forest dozens of times is expensive. Worse, the features that matter most are often those that change the treatment effect, called effect modifiers, and their signal is small next to the features that simply predict the outcome. Ordinary RFE ranks features by how well they predict the outcome, so it discards effect modifiers early.

We propose to run feature elimination as a multi-fidelity search. Multi-fidelity methods are the family behind Hyperband and successive halving: many candidates are scored cheaply, on a data subsample or with few training epochs, and only the survivors get expensive evaluations. We keep a small population of candidate feature subsets instead of one greedy path. Each subset is scored with a treatment-effect-specific criterion, and cheap evaluations are promoted to expensive ones only when the cheap evidence is statistically strong enough to trust. The central claim is that successive halving can be applied directly to the elimination path of a causal model, but only if two changes are made. First, the promotion rule must be tied to the noise level of treatment effect estimates rather than to a fixed pruning fraction. Second, a feature's value must be scored so that its roles as confounder, outcome predictor, and effect modifier cannot drown one another out.

What is new relative to prior work

Three strands of prior work touch this problem, and none does what we propose.

The first strand is causal feature selection. High-Dimensional Feature Selection for Sample Efficient Treatment Effect Estimation proves that a good covariate subset must keep both confounders, which remove bias, and pure outcome predictors, which reduce variance. It selects these with joint sparse penalties in linear models. DRCFS: Doubly Robust Causal Feature Selection tests features for direct causal effects on the outcome, which selects the parents of the outcome rather than the modifiers of a treatment effect. Feature selection for discovering distributional treatment effect modifiers targets modifiers of the whole outcome distribution through hypothesis tests. These papers motivate our scoring rule because they show that selection must be role-aware. None of them is iterative, none is computed with nonlinear effect models, and none addresses compute.

The second strand is multi-fidelity optimization, which supplies our search mechanism. It includes BOHB, DEHB, PriMO's multi-objective scheduler (Multi-objective Hyperparameter Optimization in the Age of Deep Learning), and the proxy-subset evaluation of LLM judges (Tuning LLM Judge Design Decisions for 1/1000 of the Cost). All of them allocate budget over hyperparameter or prompt configurations, where cheap and expensive scores tend to agree in rank. We move the budget allocation onto the feature elimination trajectory. There, cheap and expensive scores may not agree, because treatment effects are weak signals. That disagreement, a rank inversion, is precisely what we measure and design around.

The third strand is budgeted causal estimation. Budgeted Heterogeneous Treatment Effect Estimation and Sample Constrained Treatment Effect Estimation decide which individuals to measure. We take the data as given and decide how much computation each candidate feature set deserves. The two kinds of budget are complementary, and we do not claim to unify them here.

Multi-objective reward shaping enters in a narrower role, as a lesson about combining signals. Work on robot policy search (Redundancy Resolution as Action Bias in Policy Search for Robotic Manipulation) shows that in a weighted sum of objectives, strong terms swamp weak ones and weak terms get ignored. Optimizing Language Models with Fair and Stable Reward Composition in Reinforcement Learning answers this with max-min composition. Feature importance in a treatment effect model has the same structure, since a feature's usefulness is a sum over roles in which the outcome-prediction role dominates. We borrow the max-min idea for the retention score.

Core mechanism

We score each feature subset with a treatment-effect loss rather than an outcome loss. On held-out folds we compute the doubly robust pseudo-outcome, a per-unit quantity whose average equals the true treatment effect when either the outcome model or the treatment-assignment model is correct:

$$\tilde\tau_i = \hat\mu_1(x_i) - \hat\mu_0(x_i) + \frac{W_i - \hat e(x_i)}{\hat e(x_i)\,(1-\hat e(x_i))}\big(Y_i - \hat\mu_{W_i}(x_i)\big).$$

Here $x_i$ are unit $i$'s covariates, $W_i \in \{0,1\}$ is its treatment, $Y_i$ its observed outcome, $\hat\mu_w$ the fitted outcome model under treatment $w$, and $\hat e$ the fitted probability of being treated. The score of subset $S$ is the mean squared error between a CATE model trained on $S$ and $\tilde\tau$. It can be swapped for the estimated value of the policy that treats units whose predicted effect is positive.

Within each subset, pruning uses a role-balanced retention score. For feature $j$ we compute three importances. The confounding importance $I^{c}_j$ measures what $j$ contributes to $\hat e$. The prognostic importance $I^{p}_j$ measures its contribution to $\hat\mu_0$. The modifying importance $I^{m}_j$ measures its contribution to $\hat\mu_1 - \hat\mu_0$, estimated by permutation on the pseudo-outcome loss. The feature is retained according to

$$s_j = \max_{r \in \{c,p,m\}} \frac{I^r_j}{\operatorname{median}_k I^r_k}.$$

Normalizing within each role and taking the maximum means a feature survives if it is strong in any role. A summed score would, in effect, only keep good outcome predictors. This directly encodes the finding from High-Dimensional Feature Selection for Sample Efficient Treatment Effect Estimation that both confounders and prognostic features must be kept, and it adds protection for modifiers.

The search keeps a population of subsets. At each round, each survivor branches into children by dropping blocks of low-$s_j$ features under a few different drop sizes and random tie-breaks. This softens the irreversibility of greedy RFE. Children are scored at a low fidelity, meaning a data fraction $n_b$ and a reduced epoch count. The key departure from standard successive halving is the promotion rule. The difference in scores between two subsets, estimated on $n_b$ units, has standard error of roughly $\sigma/\sqrt{n_b}$ by the central limit theorem, where $\sigma$ is the spread of per-unit pseudo-outcome loss differences. We therefore eliminate a child only when its score gap to the current best exceeds a confidence bound at that fidelity. Candidates that cannot yet be separated are promoted to a larger $n_b$, not discarded. This predicts that a modifier with true score contribution $\delta$ can only be resolved once $n_b \gtrsim \sigma^2/\delta^2$. The prediction can be checked directly, and it tells a practitioner when cheap elimination is safe.

Research program

The work has three parts. The first is an empirical map of rank agreement. For each benchmark we compute the Spearman correlation between low-fidelity and full-fidelity subset scores, separately for subsets that differ by a confounder, a prognostic feature, or a modifier, and we test whether modifiers show the predicted rank inversion. The second is the algorithm itself, built on two model families: Dragonnet (Adapting Neural Networks for the Estimation of Treatment Effects), whose three heads give the three role importances naturally, and a gradient-boosted doubly robust learner. The third is an analysis of the confidence-bound promotion rule. We state conditions under which it keeps every modifier whose contribution exceeds $\delta$ with high probability, and we compare its compute to full-fidelity RFE as a function of $p$.

To create a known ground truth in high dimensions, we augment IHDP (25 covariates, 747 units), Twins (48 features, 11,984 pairs), and the IBM benchmark (100,000 units) with planted features. These are noise features, correlated copies of real covariates, and synthetic modifiers whose effect-scale $\delta$ we control, bringing $p$ to between 500 and 5,000. ACIC 2018's 63 data-generating processes are used without augmentation to test robustness across structures.

Baselines

The naive reference is no selection: Dragonnet with targeted regularization, trained on all features. On unaugmented IHDP, Adapting Neural Networks for the Estimation of Treatment Effects reports ATE error $\Delta_{all} = 0.11$, and on ACIC it reports a mean absolute error of 0.35. We reproduce both before augmenting. The proposal must match these on the original data and beat them in root PEHE, the root mean squared error of individual effect estimates, once noise features are added. No reported PEHE result found on Lacuna for feature-selected CATE models on augmented IHDP, so this baseline is run fresh.

The strongest existing selection method is the joint sparse estimator of High-Dimensional Feature Selection for Sample Efficient Treatment Effect Estimation. On IHDP it reports an ATE of 4.49 against a true 4.36 using an average of 6.4 of 25 features. We run it on identical augmented data. Its penalty strength gets the same tuning budget, measured in model fits, that our search gets. Beating it means lower root PEHE and higher recall of planted nonlinear modifiers at equal or fewer selected features. We expect it to be competitive when effects are linear, and we report that regime honestly.

The compute-matched competitor is full-fidelity RFE. It uses the same role-balanced score and the same Dragonnet, retrains on all data at every step, and follows one greedy path. It isolates the value of multi-fidelity scheduling. We must match its root PEHE and modifier recall at a fraction of its model-fit cost; a factor of at least five fewer full-data-equivalent fits at $p = 1{,}000$ is our target.

Three ablations isolate the key ingredients. The first replaces the confidence-bound promotion with fixed-fraction successive halving, which tests whether causal-aware promotion is what prevents modifier loss. The second replaces the max-role score $s_j$ with a summed importance and with outcome-only predictive RFE, which tests the role-balancing claim. The third sets the population size to one, which tests whether keeping several subsets beats greedy elimination.

For the budgeted setting on IBM we also report QHTE's reported $\sqrt{\text{PEHE}} \approx 19.6$ at 1,500 queries (Budgeted Heterogeneous Treatment Effect Estimation). This is context only, since it spends a different budget.

Why not just…?

Why not just fit an L1-penalized CATE learner once and skip elimination? A single sparse fit is cheap and is the most serious threat on linear problems. However, lasso on outcomes suffers regularization bias against treatment-by-covariate interactions. Mean-based selection also missed variance modifiers almost entirely in Feature selection for discovering distributional treatment effect modifiers, where true positive rates for such baselines were near zero. Our planted-modifier experiments vary linear versus nonlinear modification. If the one-shot sparse learner matches us on nonlinear modifiers, we will conclude the search is unnecessary and will report the rank-agreement map as the main contribution.

Why not just compute permutation importance once from a full Dragonnet and cut at a threshold? This is the real compute competitor, because it costs one training run. Its weakness is collinearity. When correlated copies split importance, both look weak and both are cut, which is the failure RFE's retraining was designed to fix (Multiclass cancer diagnosis using tumor gene expression signatures). The planted correlated-copy features test this directly. If one-shot importance recovers modifiers as well as our method, the retraining loop is not earning its cost.

Why not just run full-fidelity RFE and pay the compute? On IHDP-sized data this is feasible and may be the right choice, so we report the crossover point in $p$ and $n$ where it stops being affordable. Our claim is limited to the regime beyond that point, and to showing that the confidence-bound rule loses nothing relative to full fidelity.

Why not just use standard Hyperband over a "number of features" hyperparameter? Hyperband with a fixed pruning fraction is exactly what our first ablation runs. If the rank-inversion hypothesis is right, it will discard subsets containing weak modifiers at low fidelity. If it is wrong, the ablation will match our method, and we will drop the causal promotion rule.

Decisive checks

Three results decide the proposal. On augmented IHDP and Twins, at planted modifier strengths $\delta$ below the low-fidelity resolution threshold, fixed-fraction successive halving should lose modifiers that our method keeps. The loss should appear as lower modifier recall and higher root PEHE, at roughly the $n_b \approx \sigma^2/\delta^2$ crossover the analysis predicts. Across ACIC 2018's 63 processes, our method should match full-fidelity RFE's root PEHE within its standard error while using several times fewer full-data fits. On IBM at $p$ in the thousands, it should be the only role-aware nonlinear eliminator that finishes within the budget full-fidelity RFE needs for a few hundred features. We will also report the value of the resulting treatment policies, since Machine learning who to nudge shows that predictive and causal targeting can diverge sharply in realized gains. If the first check fails, meaning cheap scores rank modifier-containing subsets correctly after all, the method simplifies to plain successive halving. That would be a useful negative result.