Choosing Propensity-Model Complexity by Causal Error, Not Prediction Error

In a retrospective cohort study, researchers use historical health records to estimate whether a treatment helped. Treated and untreated patients usually differ, so the standard correction first fits a propensity model, a classifier that predicts which patients received treatment, and then reweights or matches patients so the two groups look alike. Analysts almost always choose that model's complexity the way any classifier's complexity is chosen: by cross-validated accuracy at predicting treatment. That target is the wrong one. A propensity model that predicts treatment too well picks up each hospital's prescribing habits. It pushes predicted probabilities toward 0 and 1, produces enormous weights, and gives effect estimates that are both unstable and biased. The tension between fitting the training data and generalizing to new data, familiar from learning theory, reappears here with a different meaning for "generalizing." What matters is whether the effect estimate is right, and in particular whether it stays right when the analysis moves to another hospital.

We propose to choose propensity-model complexity directly by a proxy for causal error. The proxy is negative control outcomes: outcomes the treatment cannot plausibly affect, such as ingrown toenails for an anticoagulant, so any estimated effect on them is bias. We measure that bias on hospitals or data eras held out from model fitting. The claim is concrete and testable. The complexity that minimizes treatment-prediction loss is systematically higher than the complexity that minimizes causal error, and the gap widens when the target site differs from the training sites. If the claim holds, the fix is simple, cheap, and can be audited: a different rule for picking the regularization strength or the number of boosting rounds.

Core Claim and What Is New

Three bodies of work in the file circle this idea without reaching it.

The first supplies the evaluation method. Evaluating the impact of instrumental variables in propensity score models uses 49 negative controls in a 77,000-patient MarketScan claims cohort to compare 33 fixed propensity specifications. It finds that an L1-regularized logistic model over all 52,000+ covariates reaches 84% coverage of the null, against 71% for an outcome-selected covariate set. That paper treats negative controls only as a final scorecard. Its regularization strength is chosen in the conventional way, and it never asks whether the negative-control signal should pick the regularization strength itself, or whether its conclusions survive a change of site.

The second supplies the motivation. Improved Outcome Prediction Across Data Sources Through Robust Parameter Tuning shows across 15 microarray studies that internal cross-validation under-regularizes Lasso and boosting, and that deliberately sparser models chosen against external data transfer better. It is purely predictive, with no treatment, no confounding and no counterfactual. It also warns that tuning and evaluating on the same external set inflates results, a "winner's curse" our protocol must avoid. Generalizing Across Domains in Diabetic Retinopathy via Variational Autoencoders adds two observations: a larger backbone hurt cross-clinic accuracy by 1.5%, and tuning on the target clinic ("target peeking") invalidates domain-generalization claims.

The third constrains the design. Matching on the Estimated Propensity Score proves that estimating the propensity score helps average-treatment-effect efficiency, but only for low-dimensional parametric models. A Tutorial for Propensity Score Weighting Methods Under Violations of the Positivity Assumption documents how extreme scores break weighting. Together they explain why the classical guarantee cannot simply be carried over to high-capacity models.

The new element is therefore the selection signal. We pick complexity by squared negative-control estimates on held-out sites, and we keep the negative controls used for tuning separate from those used for evaluation. We also map the resulting causal-error curve against the prediction-loss curve, which has not been done. We deliberately do not build a new estimator. Reliable Estimation of Individual Treatment Effect with Causal Information Bottleneck penalizes representation complexity with mutual-information estimators. Structural Risk Minimization for Learning Nonlinear Dynamics shows that a capacity bound can locate the bottom of a U-shaped test-error curve that training error misses. Both inform the framing, but the file gives us no reason to think a new architecture is the bottleneck. Choosing complexity looks like the cheaper and more decisive question.

Research Program

The procedure works on a nested family of propensity models indexed by one complexity knob $\lambda$. For L1 logistic regression the knob is the penalty strength. For gradient boosting it is the number of rounds, and for a small neural network it is width combined with weight decay. For each value of $\lambda$ we fit the propensity model on all training sites except one. On the held-out site $s$ we estimate the effect of treatment on each tuning negative control $k$, using one fixed downstream estimator, and score

$$\mathrm{NC}(\lambda)=\frac{1}{|S|\,|K_{\text{tune}}|}\sum_{s\in S}\sum_{k\in K_{\text{tune}}}\hat\theta_{k,s}(\lambda)^2 .$$

Here $\hat\theta_{k,s}(\lambda)$ is the estimated log hazard ratio for negative control $k$ at held-out site $s$. Its true value is zero, so its square combines bias and variance. $S$ is the set of held-out sites, and $K_{\text{tune}}$ is the subset of negative controls reserved for tuning. Because extreme weights inflate variance, the criterion penalizes collapse of the overlap between treated and untreated patients (a positivity violation) without a separate trimming rule. The selected $\lambda^\ast$ is then refit on all training sites and evaluated on a final test site with a disjoint set of evaluation negative controls $K_{\text{eval}}$, using a nested leave-one-site-out protocol so the final site never influences tuning.

The analytical deliverable is the pair of curves, propensity log-loss and $\mathrm{NC}$, traced across $\lambda$. We report where each curve bottoms out, and how far apart the two minima sit in same-site versus cross-site evaluation. Alongside the curves we report effective sample size and maximum weight at each $\lambda$, so the mechanism is visible: does the causal-error curve turn upward at the point where weights explode?

Data and Ground Truth

The file offers three tiers of data. MIMIC-III (MIMIC-III, a freely accessible critical care database) is public and single-center, but its switch from the CareVue to the MetaVision charting system creates a natural era shift in how data were recorded. We use the two eras as pseudo-sites, with an exposure such as early versus late mechanical ventilation, following the logic of Sternlicht et al.. The MarketScan anticoagulant cohort with its 49 published negative controls is the second tier. The third is the 375-hospital stroke registry (Risk factors associated with 31-day unplanned readmission), which provides genuine multi-hospital shift but depends on collaborator access.

Because negative controls only detect bias at a true effect of zero, we add plasmode simulations on each real covariate structure. A plasmode keeps the real patients' covariates and treatments but simulates outcomes with known, non-zero effects. This gives true bias and RMSE at non-null effects. IHDP and ACIC 2016 serve as public sanity checks, though their small size and single-site structure cannot test the cross-site part of the claim.

Baselines

The naive reference is an unweighted comparison of treated and untreated patients, together with a clinician-curated low-dimensional logistic propensity model of the kind used in Long-Term Survival in Men With High Grade Prostate Cancer. These anchor how much any adjustment helps.

The strongest existing method on this question is the large-scale L1 logistic propensity model over all covariates, with penalty chosen by internal cross-validated log-loss, from Evaluating the impact of instrumental variables. It reports 84% coverage of the null across 49 negative controls on MarketScan. It is the strongest competitor because it is current large-scale practice and has already beaten variable-selection alternatives on exactly this metric. We rerun it on identical cohorts with the same covariate set, the same downstream estimator, and the same $\lambda$ grid. To match compute, it gets the same number of model fits as our search. Beating it means higher coverage and lower mean squared error on the evaluation negative controls at the held-out site, together with lower plasmode RMSE. The coverage gain must exceed the bootstrap interval of the difference across controls.

We also run the robust-tuning rules Robust1SE and RobustTuneC from Improved Outcome Prediction Across Data Sources Through Robust Parameter Tuning, applied to propensity log-loss. They test whether simply underfitting more is enough, without a causal signal. That paper reports results only as rank plots, so there is no reported result found on Lacuna for a numeric baseline value.

As a modern flexible estimator we include CausalPFN, a transformer pretrained on synthetic causal problems that estimates effects in a single forward pass, run on subsampled cohorts to fit its context window. It reports best median PEHE rank across 310 tasks but no IHDP number on Lacuna. For context on IHDP, Causal Information Bottleneck reports an out-of-sample $\sqrt{\epsilon_{PEHE}}$ of 0.613.

Two ablations isolate the key ingredient. In one, the same negative-control criterion is computed in-site, so causal signal is present but there is no site shift. In the other, log-loss is computed on held-out sites, so site shift is present but there is no causal signal. Our method should beat both. If held-out log-loss matches it, the causal signal is unnecessary. If in-site negative-control scoring matches it, the cross-site part is unnecessary.

Why not just…?

Why not just tune for covariate balance, as many practitioners already do? This is the most serious threat. Balance diagnostics are cheap and causally motivated. However, they are computed on the same patients the model was fit to and only check moments of measured covariates. An overfit propensity model can achieve excellent in-sample balance while its weights have exploded, so balance says nothing about variance or transfer. We include balance-based stopping, which selects the $\lambda$ minimizing the maximum standardized mean difference, as an additional baseline under the same budget. If it matches our criterion on held-out evaluation controls and plasmode RMSE, we will report that balance suffices, and the contribution becomes a validation of balance tuning across sites.

Why not just use overlap weights or trim extreme scores? The positivity tutorial (A Tutorial for Propensity Score Weighting Methods Under Violations of the Positivity Assumption) shows these remedies stabilize weighting, and they may remove much of the damage from overfitting. Trimming, though, changes the population the effect describes, and it does not fix bias from site-specific prescribing patterns learned by the model. We cross every tuning rule with plain inverse-probability, trimmed, and overlap weighting. If the advantage of negative-control tuning disappears under overlap weights, we say so plainly.

Why not just pick the simplest model within one standard error (Robust1SE)? This tests whether a fixed tilt toward simplicity captures the effect. The predictive evidence (Improved Outcome Prediction Across Data Sources Through Robust Parameter Tuning) says such a tilt helps, but the size of the tilt it needs depends on how severe the shift is. That is why the paper's adaptive RobustTuneC needed external data. Our curves show whether the causal optimum sits a consistent distance below the log-loss optimum. If it does, a fixed rule is the right recommendation.

Decisive Checks

The central experiment is the curve comparison on MIMIC-III across the CareVue/MetaVision boundary and on MarketScan, repeated on the stroke registry if access allows. The claim is supported if the $\mathrm{NC}$-optimal $\lambda$ is consistently more regularized than the log-loss optimum, with a larger gap across sites than within a site. It also requires that the selected models beat internal-CV L1 on held-out evaluation controls and plasmode RMSE.

The claim is refuted if the two minima coincide, or if the held-out log-loss ablation matches our method. It is also refuted if negative-control scores are too noisy to rank $\lambda$ values. We detect that by reselecting $\lambda^\ast$ with bootstrap resamples of $K_{\text{tune}}$ and checking whether the selection is stable. Finally, we check that low negative-control bias predicts low plasmode bias at non-null effects. Without that link, a good negative-control score would not certify the effect estimates researchers actually care about.