Clinical machine learning is moving from predicting outcomes, such as "will this patient die," toward predicting interventions, such as "will this patient be put on a ventilator in the next few hours." That shift is the response to the prediction-intervention gap: an accurate risk score does not tell a clinician what to do. Once a model predicts clinician actions, its outputs become inputs to causal estimation. The predicted probability that a patient receives a treatment, called the propensity, determines the reweighting used to estimate what would have happened under a different treatment policy. Two pressures then pull against each other. Treatment onsets are rare, so practitioners upweight them in the training loss (weighted cross-entropy) to keep the model from ignoring them. Causal estimators, however, need those probabilities to be honest, and upweighting inflates them.
This note proposes a controlled study of a question the literature has not separated cleanly. When a shared multi-task network predicts both patient outcomes and rare clinician interventions, does class weighting help or hurt downstream treatment-effect and policy-value estimates, and is any effect due to the network's internal features or only to its output scale? The output-scale part can be undone exactly with a closed-form logit shift. What remains after that correction is the real effect of weighting on the learned features. Our hypothesis is that this remaining effect is not zero, that its sign can be predicted from a simple geometric property of the learned embedding, and that it matters most where causal estimation is most fragile: in the low-overlap regions where treatment is rare.
Core Claim
The research file frames the conflict as calibration versus detection. Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters shows that a behavior-policy error of only 0.06 can produce over 50% error in an off-policy value estimate, and that neural estimators are badly calibrated in severe, data-sparse states. Weighted cross-entropy seems to make this worse. Read that way, though, the conflict is partly illusory. At the population optimum, a classifier trained with positive weight $w_1$ and negative weight $w_0$ learns shifted log-odds:
$$\text{logit}\,\hat p_w(x) = \text{logit}\,\eta(x) + \log\frac{w_1}{w_0},$$
where $\eta(x)$ is the true treatment probability given covariates $x$. Subtracting $\log(w_1/w_0)$ from the logit, which we call prior correction, removes the distortion at the head.
The open question is therefore not whether weighting miscalibrates the head. It does, and the fix is known. The open question is whether weighting changes the shared representation in ways that survive correction. When resampling/reweighting improves feature learning in imbalanced classification? A toy-model study supplies the theory that makes this a sharp question. In a linear Gaussian model, reweighting strictly degrades feature alignment when the two classes have equal variance. It helps only when they differ, and specifically when the minority class has the smaller variance and so carries more information about the boundary.
Our claim is that patients about to receive an intervention plausibly form a tighter cluster in embedding space than the heterogeneous mass of untreated patient-hours. If so, weighting should improve the features used for propensity estimation in exactly the region where overlap vanishes. Prior correction should then turn that improvement into lower error in causal estimates. If the variance ratio runs the other way, weighting should hurt. This prediction can be checked per task before looking at downstream error.
Research Program
The base system is a multi-task recurrent network built on hourly MIMIC-III time series from MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III. It uses the gap-time windowing of Clinical Intervention Prediction and Understanding with Deep Neural Networks so that predictions arrive early enough to act on.
A shared encoder feeds two heads. The outcome head predicts 24-hour decompensation, in the style of Multitask Learning and Benchmarking with Clinical Time Series Data. The treatment head predicts vasopressor or ventilation onset, and its corrected output serves as the propensity model. This pairing follows the argument of On the Actionability of Outcome Prediction: outcome risk alone has little action value unless one action dominates for everyone. A useful system must therefore model the action and its effect, not only the risk. The multi-task structure here is a means of sharing features between risk and treatment assignment. It is not the object of study.
The manipulated variable is where class weighting enters the network. The levels are no weighting; weighting applied end to end through the shared encoder; and decoupled weighting. In the decoupled condition, the encoder trains on unweighted losses and only the treatment head is retrained with weights, following the decoupling practice that the Obuchi et al. paper cites from long-tailed recognition. Every condition then receives the same prior correction, followed by the same temperature scaling on a held-out split. Any remaining differences are therefore attributable to the representation.
For each condition we measure the class-conditional covariance of the encoder's embeddings for treated and untreated hours. We summarize it as the ratio of their traces projected onto the head's discriminative direction. This is the empirical analogue of the variance asymmetry in the Gaussian toy model, and it is the quantity whose sign we predict will determine whether weighting helps.
Task weighting across the two heads uses Loss-Balanced Task Weighting from Loss-Balanced Task Weighting to Reduce Negative Transfer in Multi-Task Learning, which scales each task's loss by $(\mathcal{L}_k(t)/\mathcal{L}_k(0))^\alpha$. The research file worries that class weighting will distort this schedule. A fixed multiplicative rescaling of a task's loss cancels in that ratio. Class weighting is not a pure rescaling, however, because it changes the mix of easy negatives and hard positives over training. We will log the schedule under each weighting condition and report whether it differs. We do not assume an interaction.
Evaluation needs ground truth for counterfactuals, which observational logs cannot provide. We will build a semi-synthetic benchmark in the spirit of the one in Overlap-weighted orthogonal meta-learner for treatment effect estimation over time. Real MIMIC-Extract covariate trajectories are kept. Treatment onset is drawn from a known logistic function of those covariates, calibrated to hourly onset rates of roughly 1–5%. Outcomes are drawn from a known model with heterogeneous treatment effects.
We will vary two things in the simulator: the onset rate, and whether the treated covariate distribution is tighter or looser than the untreated one. This lets us cross our manipulation with the exact condition the toy theory says should flip the sign. Because the true propensity is known, we can measure propensity error against the truth directly rather than only through empirical calibration curves. The downstream estimators are held fixed across all conditions: inverse-propensity weighting, a doubly robust learner, and the overlap-weighted orthogonal learner. Only the propensity input changes.
Baselines
The naive reference point is a single-task logistic regression propensity model trained with unweighted cross-entropy, fed to the same downstream estimators. This is what a careful applied analyst would do by default, and the multi-task benchmark reports logistic regression as a respectable baseline: in-hospital mortality AUC-ROC of 0.848 against 0.870 for the multitask channel-wise LSTM in Multitask Learning and Benchmarking with Clinical Time Series Data.
The strongest detection baseline is a single-task random forest per intervention. MIMIC-Extract reports macro AUROC of 0.90 to 0.94 for intervention onset and offset, with AUPRC "often below 0.50." It receives the same features, the same patient-level splits, and its own hyperparameter search at a matched number of trials.
The strongest calibration baseline is the approximate nearest-neighbor behavior-policy estimator of Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters. On the MIMIC-III sepsis cohort it reports a total variation distance of 0.199 against 0.426 for a neural network in the most severe states, and off-policy value MSE of 2.48 against 4.04. Those numbers come from a different cohort and estimator, so they serve as a bar for the size of the gap rather than a target to match. We will run nearest neighbors both on hand-built physiological features, as in that paper, and on our unweighted encoder's embeddings. The second version separates the value of the learned features from the value of non-parametric estimation.
On the semi-synthetic causal benchmark itself we found no reported numerical result on Lacuna. The overlap-weighted learner paper reports up to 58.4% relative error reduction on its fully synthetic low-overlap data but quotes no figure for the MIMIC-III semi-synthetic cohort. We will therefore establish all baselines ourselves under identical simulators.
The ablation that isolates the key ingredient is the comparison between end-to-end weighting and decoupled weighting, both after prior correction and temperature scaling. If they match, weighting has no representational effect and the whole question reduces to head correction. The proposal counts as beating the baselines if the best weighting placement, chosen by the variance-ratio rule rather than by test error, lowers policy-value or treatment-effect error relative to all three baselines. That reduction must exceed seed-to-seed variation across at least five seeds, and it must be concentrated in the lowest-overlap tenth of patient-hours.
Why not just…?
Why not just train unweighted and apply temperature scaling? This is the most serious threat, and it is a full arm of the study. If unweighted training plus temperature scaling matches every weighted variant on downstream error, the conclusion is that weighting buys detection at the head and nothing in the features. That would be a useful negative result for clinical causal pipelines. The Obuchi et al. analysis predicts that outcome only when class variances are equal. The simulator's variance-asymmetry axis is designed to show whether real-looking clinical data lies in that regime or not.
Why not just use decoupled training, as the long-tailed recognition literature recommends? Decoupling is a direct competitor and could win. In the toy theory it is the right choice exactly when the classes have equal variance. What the theory leaves open is whether end-to-end weighting wins when the minority class is tighter. Our experiment crosses placement with variance ratio, so it will show whether "always decouple" is the right default or whether the choice should depend on a measurable embedding property.
Why not just use nearest neighbors for the propensity, given how well they calibrate? Raghu et al. show nearest neighbors calibrate better than neural networks on sepsis data. They also note that nearest neighbors depend on hand-built distance metrics and become expensive at scale. Running nearest neighbors on our learned embeddings tests whether the representation is the bottleneck. If nearest neighbors on hand-built features still win after our correction, then neural propensity models should not be used in this setting, and we will report that.
Why not just train separate single-task models and skip multi-task learning? Loss-Balanced Task Weighting documents negative transfer on 48 of 128 tasks under plain multi-task training, and random forests are competitive on MIMIC-Extract interventions. We therefore include a single-task treatment-only encoder under every weighting placement. If multi-task sharing does not improve propensity error in low-overlap regions, the multi-task component drops out of the recommendation. The weighting question stands independently of it.
Decisive Checks
The first check is mechanistic. Across simulator settings, the sign of the corrected downstream benefit of end-to-end over decoupled weighting should follow the sign of the measured embedding variance ratio, as the Gaussian toy model predicts. If the two are unrelated, the theoretical link fails, even if some weighting scheme happens to win.
The second check is about where gains appear. Improvements in propensity error against the known truth should concentrate in low-overlap patient-hours, and they should carry through to lower error from inverse-propensity weighting and smaller but nonzero gains for the doubly robust and overlap-weighted estimators. The more robust estimators should be less sensitive to propensity error by design, so a gain that shows up only for them would point to an artifact.
The third check is on real data. On real MIMIC-Extract onset tasks we report AUPRC, expected calibration error after correction, and the measured variance ratio. We also apply the placement rule chosen on the simulator to real vasopressor and ventilation onset, to see whether the semi-synthetic finding transfers. Real-data causal claims are limited to agreement in sign, because the counterfactual ground truth exists only in simulation.
References
- Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters.
- When resampling/reweighting improves feature learning in imbalanced classification? A toy-model study
- MIMIC-Extract: a data extraction, preprocessing, and representation pipeline for MIMIC-III
- Clinical Intervention Prediction and Understanding with Deep Neural Networks
- Multitask Learning and Benchmarking with Clinical Time Series Data
- On the Actionability of Outcome Prediction
- Loss-Balanced Task Weighting to Reduce Negative Transfer in Multi-Task Learning
- Overlap-weighted orthogonal meta-learner for treatment effect estimation over time