Outcome-Side Forest Screening for Model-Averaged Propensity Scores in Wide EHR Cohorts

Retrospective cohort studies estimate whether a treatment helps by mining historical hospital records instead of running a trial. To compare treated and untreated patients fairly, they adjust for patient characteristics that could explain both who got treated and how they fared. The usual adjustment tool is a propensity score, a model of each patient's probability of being treated. One recent method makes that score much more robust. It averages several propensity models (logistic regression, single trees, random forests, boosting) and weights them by how well they balance patients' predicted untreated outcome, rather than by how well they predict treatment. That method was only tested with about ten covariates. Real electronic health records offer hundreds to thousands of candidate variables. Many of them drive treatment choice without affecting the outcome, and adjusting for those makes estimates noisier without making them less biased.

We propose a screening step that sits in front of that averaged propensity score. A random forest is fit to predict the outcome among untreated patients, and its feature importances are scored out of sample using MDI+. A second forest predicts treatment and is used only to flag variables that predict treatment but carry no outcome signal; those are discarded. Every stage runs on separate data folds, so the variables chosen never touch the data used to estimate the effect. The claim to test is specific. In wide, nonlinear cohorts, this outcome-side, out-of-sample forest screening should give treatment-effect estimates with lower error and valid confidence intervals. It should do better than four alternatives: screening with the forest's built-in importance scores, double-selection with a linear penalized regression, skipping selection and using a doubly robust estimator on everything, and logistic propensity matching.

Core Claim

Model averaging and feature selection have each been shown to help causal estimation, but under incompatible assumptions. Prognostic score-based model averaging approach for propensity score estimation shows that weighting a pool of propensity models by balance on the prognostic score (the predicted outcome under no treatment, learned from untreated patients) keeps 95% interval coverage above 90%, where single-model methods fall below 30%, in nonlinear simulations. Its authors state that real data needs a high-dimensional front end, and they did not split samples between stages. High-Dimensional Feature Selection for Sample Efficient Treatment Effect Estimation shows which variables to keep: confounders and outcome-only predictors. It also shows that keeping treatment-only drivers inflates variance. Its selector, however, assumes a linear outcome model.

Our claim is that these two results can be joined in one way that does not discard the nonlinearity the averaging was built to handle. The screening criterion should be an out-of-sample importance for the outcome model, fit on untreated patients with a tree ensemble that can also represent smooth trends. The treatment model's importance should act only as a veto signal.

The out-of-sample requirement matters in electronic health records specifically. Standard impurity importance (the total drop in split error credited to a feature) favors continuous and high-cardinality variables because they offer more split points. Health records mix continuous labs with thousands of binary codes, so impurity importance should preferentially admit continuous variables that drive treatment, such as a lab that triggers a protocol. MDI+ (MDI+: A Flexible Random Forest-Based Feature Importance Framework) replaces the in-sample score with a leave-one-out score on a regularized linear model built from the forest's splits plus the raw features. It should therefore remove this bias.

Concretely, let $I_Y(k)$ be the MDI+ importance of feature $k$ in an outcome model fit on untreated patients in a held-out fold. Let $I_T(k)$ be the same quantity for a treatment model. Let $\tau_Y, \tau_T$ be the 95th percentiles of the corresponding importances among permuted "shadow" copies of the features, which serve as a null reference. The adjustment set is

$$S = \{k : I_Y(k) > \tau_Y\},$$

and $\{k : I_T(k) > \tau_T,\ I_Y(k) \le \tau_Y\}$ is recorded as the putative treatment-driver set and excluded. The treatment model is never used to add variables, because doing so is the route by which treatment drivers enter.

Relation to Prior Work

Three groups of work frame the proposal.

The first group motivates it by documenting the gap. Retrospective EHR studies such as Predicting pressure injury risk in hospitalised patients using machine learning with electronic health records: a US multilevel cohort study screen 40 features down to 20 with a forest. Machine Learning Prediction Models for Mechanically Ventilated Patients: Analyses of the MIMIC-III Database ranks variables with tree importance and SHAP. Both serve prediction, and neither separates treatment drivers from confounders. We use forest screening for causal adjustment and draw exactly that separation.

The second group supplies the methods we reuse. We take the prognostic-score model averaging estimator unchanged as the back end. We take RF+ and MDI+ as the screening engine. We take the variable-role taxonomy (treatment drivers, confounders, outcome-only predictors) from the high-dimensional selection paper. Unlike that paper, our selector is nonlinear. Unlike the averaging paper, ours scales to thousands of covariates with strict sample splitting.

The third group constrains the design. A Machine Learning Approach to Predicting Heterogeneous Lipid Responses to Statins in Type 2 Diabetes found that Lasso beat random forests on smooth lipid trends (LDL-C $R^2 = 0.520$). A plain forest screen would therefore miss smooth confounders, which is why we use RF+ with raw linear features rather than a plain forest. Orthogonal Random Forest for Heterogeneous Treatment Effect Estimation shows that orthogonalized two-stage forests can handle high-dimensional confounding without explicit selection. That is the strongest argument that our screening step is unnecessary, and we test it head on rather than assume it away.

Research Program

The pipeline uses three-way cross-fitting. Fold A fits the outcome forest on untreated patients and the treatment forest on everyone, computes MDI+ and shadow-feature thresholds, and outputs $S$. Fold B fits the candidate propensity models (logistic regression, single tree, random forest, boosting) and the averaged prognostic score on $S$, then solves the averaging weights by prognostic-score balance. Fold C applies the resulting weights and estimates the average treatment effect. Folds rotate, and estimates are pooled. Confidence intervals come from a bootstrap over the whole pipeline, including selection, so that selection uncertainty is counted.

We also measure selection stability across folds, using the overlap of selected sets between folds. Instability under correlated variables is a known weakness of forest rankings, and MDI+ is reported to be more stable on TCGA.

The analytic part asks a narrow question: when does impurity bias translate into causal error? In the semi-synthetic design we vary the type of the treatment drivers (continuous versus binary). This tests the predicted mechanism directly. The gap between in-sample impurity screening and MDI+ screening should be large when treatment drivers are continuous and near zero when they are binary.

Baselines

The naive reference is logistic-regression propensity matching on all available covariates, the standard in the clinical literature, as in Long-Term Survival in Men With High Grade Prostate Cancer: A Comparison Between Conservative Treatment, Radiation Therapy and Radical Prostatectomy—A Propensity Scoring Approach. A second naive reference is the unadjusted difference in means.

The strongest existing selection-based competitor is the jointly sparse doubly robust estimator from High-Dimensional Feature Selection for Sample Efficient Treatment Effect Estimation. On IHDP it reports an average-effect estimate of 4.49 against a true value of 4.36, with the non-sparse version at 4.76. On Cattaneo2 it reports −239.3 g. Beating it on IHDP means an absolute error below 0.13 averaged over replications, together with nominal interval coverage. Beating it on our nonlinear semi-synthetic cohorts means lower root-mean-squared error at equal or better coverage.

The strongest selection-free competitor is the orthogonal random forest, run on all covariates with its forest-based Lasso nuisance models. The file reports no Lacuna result for it on IHDP or MIMIC-derived cohorts, so we run it ourselves. The prognostic-score averaging estimator on all covariates, without screening, is the direct parent method. Its reported coverage above 90% holds only at ten covariates, and we expect it to degrade as dimension grows.

The ablations isolate each ingredient. Replacing MDI+ with in-sample impurity importance tests the out-of-sample criterion. Replacing it with forest permutation importance tests whether any decent out-of-sample importance suffices. Adding the treatment-model features to $S$ (the union rule) tests the veto. Removing cross-fitting tests leakage.

All methods receive the same folds, the same covariate matrix, and the same hyperparameter budget: 50 random-search configurations per learner, tuned only on predictive loss within the training fold. No method is ever tuned on effect error.

Why not just…?

Why not just use permutation importance from a plain forest for screening? This is the most serious threat to the MDI+-specific claim. Permutation importance on held-out data is also out of sample, so it may remove most of the cardinality bias. MDI+ reports a gain of more than 10% in true-feature recovery AUROC over permutation importance and SHAP, but that result comes from molecular simulations, not causal screening. If our permutation-importance ablation matches MDI+ on effect error, we will report that the outcome-side, out-of-sample, cross-fitted design is the contribution and that the importance measure is interchangeable. That would be a simpler and still useful finding.

Why not just run a doubly robust or orthogonal forest on all covariates? Orthogonal estimation removes first-order sensitivity to errors in the treatment and outcome models, so it may already handle wide covariate sets. It does not remove the variance cost of adjusting for strong treatment drivers. Strong treatment drivers push propensities toward 0 or 1, and the weights explode; the high-dimensional selection paper's 4.76-versus-4.49 IHDP comparison illustrates that cost for a non-sparse estimator. Our semi-synthetic design varies the strength of treatment drivers. If the orthogonal forest matches our pipeline even when treatment drivers are strong, the screening step is not needed, and we will say so.

Why not just use linear double-selection, taking the union of variables picked by a Lasso on outcome and a Lasso on treatment? It is cheap and well understood, and Lasso beat forests on smooth lipid outcomes. It misses U-shaped and threshold confounders, such as the BMI–mortality curve in Association of body mass index with severity and mortality of COVID-19 pneumonia: a two-center, retrospective cohort study from Wuhan, China. Its union rule also deliberately admits treatment drivers. We include U-shaped confounders and purely linear confounders as separate settings. A Lasso win in the linear setting is expected and acceptable; a Lasso win in the U-shaped setting would refute our claim.

Proposed Experiments

The primary testbed is semi-synthetic but built on real covariates. We use the 104 clinical aggregates from MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III, plus binary diagnosis and procedure codes from MIMIC-III, a freely accessible critical care database, giving roughly 1,000 to 2,000 columns. Real covariate correlations and missingness are kept. Treatment and outcome are simulated with a known split of variables into treatment drivers, confounders, and outcome-only predictors. We vary dimension, the nonlinearity of confounders (linear, U-shaped, interaction), whether treatment drivers are continuous or binary, and their strength. The metrics are root-mean-squared error of the average effect, 95% interval coverage, recall of the confounder and outcome-only sets, the rate at which treatment drivers are admitted, and cross-fold selection stability.

The secondary testbeds are IHDP, where ground truth is known and the reported 4.49-versus-4.36 result is the bar, and Cattaneo2, where we check agreement with the reported −239.3 g.

The real-data demonstration follows the stability criterion of the averaging paper, whose single-model estimates ranged from −0.936 to −1.229. In a MIMIC-III treatment-timing question, such as early vasopressor initiation and 28-day mortality, we report the spread of effect estimates across candidate prognostic models with and without screening. Because the true effect is unknown, this result is supportive evidence rather than proof.

The decisive checks are three.

First, MDI+ screening must beat in-sample impurity screening specifically when treatment drivers are continuous, and not when they are binary. If the gap appears in both conditions or neither, our proposed mechanism is wrong.

Second, the full pipeline must keep coverage near 95% at 1,000 or more covariates, where unscreened prognostic averaging is expected to lose coverage.

Third, it must beat both linear double-selection and the all-covariate orthogonal forest in the nonlinear, strong-treatment-driver settings. If either of these fails, the proposal reduces to a recommendation for the simpler competitor.