When a patient's tumor shrinks on a new drug, a model that sees the scans has two very different stories to choose between. The drug may have worked, or this tumor may have been on a slow, indolent course anyway. Models that estimate who benefits from a treatment are known to pick the second kind of signal and report it as the first. That second kind of signal is a prognostic signal: it says how a patient will fare regardless of therapy. The first kind is a predictive signal: it says how much this particular treatment changes the outcome for this patient. Mixing the two up is called prognostic-predictive confounding. Longitudinal imaging makes the problem worse. The most informative prognostic cue in a scan series is how fast the disease was already changing, and models built to read image sequences are designed to pick up exactly that.
We propose a simple fix. Before treatment starts, every patient's earlier visits are an untreated record of how their disease evolves on its own. Every pair of consecutive pre-treatment scans, from treated and untreated patients alike, is a free training example of natural history. We train a forecaster on these pairs to predict each patient's untreated future. A small treatment-effect model is then fit only to the gap between the observed post-treatment outcome and that forecast. The claim is that this use of pre-treatment time, rather than any new architecture, is what lets a model separate drug response from disease velocity in image sequences. The benefit should be largest when the control arm is small and natural history dominates the outcome, which is the usual clinical case.
Core Claim
Write the outcome after treatment decision $W \in \{0,1\}$ for a patient with pre-treatment image history $H = (X_1, \dots, X_K)$, acquired at irregular times, as
$$Y = \mu_0(H) + W\,\tau(H) + \varepsilon .$$
Here $\mu_0(H)$ is the untreated (prognostic) outcome, $\tau(H)$ is the predictive treatment effect, and $\varepsilon$ is noise. This is the additive decomposition used throughout the CATE (conditional average treatment effect) literature. In imaging, $\mu_0$ carries most of the variance, and much of that variance is the rate of change visible across $H$.
Our estimator first learns a natural-history forecaster $g$ from all untreated transitions $(X_j, \Delta t, X_{j+1})$. Every patient contributes these before their treatment decision. We set $\hat\mu_0(H)$ from $g$ extrapolated to the outcome horizon, and then fit
$$\hat\tau = \arg\min_{\tau} \sum_{i:\,W_i=1} \big(Y_i - \hat\mu_0(H_i) - \tau(H_i)\big)^2 + \lambda\,\Omega(\tau).$$
In this objective, $\Omega$ is a capacity penalty that keeps $\tau$ simpler than $\mu_0$, and $\lambda$ sets its strength. The untreated patients' post-decision outcomes calibrate the gap between the forecaster's horizon and the trial outcome.
The identifying assumption is that pre-treatment dynamics are time-homogeneous given the current state and the elapsed time. Under that assumption, untreated transitions observed before the decision tell us what the untreated transition after the decision would have been. The assumption is testable on held-out untreated patients. Our benchmark also deliberately violates it to show where the method breaks.
Where This Sits
Three lines of work motivate the gap.
Enhancing Predictive Imaging Biomarker Discovery Through Treatment Effect Analysis shows that a two-headed TARNet on raw pixels can localize simulated predictive features while ignoring dominant prognostic ones. It sees only one baseline snapshot, so disease velocity never enters.
SyncTwin uses pre-treatment history to build a synthetic control for each treated patient. It recovered a statin trial's average effect of −1.26 mmol/L as −1.25 mmol/L, where deep baselines gave roughly −0.72 to −0.83. This is the closest conceptual ancestor of using pre-treatment time as control information. It works on low-dimensional lab series, matches patients rather than learning a transition model, and does not ask which features modify the effect.
Temporal radiomics for NSCLC shows that pre-to-post CT shape changes predict pathologic complete response, with external AUC 0.78. Its cohort has a single treated arm, so it cannot tell drug response from favorable natural history. It is the clinical example of the confound we target.
The CATE methodology literature constrains our design.
On Inductive Biases for Heterogeneous Treatment Effect Estimation shows that sharing prognostic structure between treatment arms, as in FlexTENet, beats independent arms. We take that lesson but move the shared structure out of the outcome network and into a forecaster trained on separate, more abundant data.
Svensson et al. show that T-learners turn small mismatches in fitting the prognostic main effect into false predictive biomarkers. This rules out training two independent sequence backbones.
Crabbé et al. show two things. Low PEHE (Precision in Estimating Heterogeneous Effects, the squared error of the estimated effect) does not mean correct attribution. Balancing penalties also erase predictive variance when treatment assignment depends on predictive features. We therefore avoid representation balancing altogether and judge methods on attribution as well as PEHE.
Really Doing Great at Estimating CATE? warns that benchmark rankings flip with the data-generating process. Our benchmark varies the process along several axes for that reason.
The longitudinal imaging literature supplies the forecaster's design.
CRONOS models scan evolution as a continuous-time flow conditioned on real-valued time gaps. It is one of few methods to beat the last-context-image heuristic, reaching 7.55 versus 8.38 NRMSE on the Lumiere glioma series. BioViL-T separates static appearance from change between scans and handles unregistered images. Neither conditions on treatment. We reuse their machinery as the natural-history model $g$ and leave causal adjustment to the second stage.
Longitudinal Causal Image Synthesis goes the other direction: it generates images from tabular interventions. It does not discover image-based effect modifiers.
Research Program
The work has three parts.
The first is a benchmark. No dataset in the file has longitudinal images with known treatment effects. We will extend the synthetic OCT generator of A Framework for Evaluating Predictive Models Using Synthetic Image Covariates and Longitudinal Data, whose code and data are released. That generator links a VAE latent space to mixed-effects disease trajectories and decodes latents into realistic scans with Stable Diffusion. We will evolve each synthetic patient's latent over several irregularly spaced visits. A drift along one latent direction sets prognostic velocity, and a second direction sets static prognostic severity. Each latent is decoded into a scan. At the decision visit, treatment is assigned and the outcome is generated with $\tau$ depending on a third, predictive direction. Because the generating directions are known, we have ground-truth attributions in the style of ITErpretability.
The benchmark has five control knobs:
- the ratio of prognostic to predictive variance;
- the size of the control arm;
- how strongly assignment depends on predictive or prognostic features;
- visit irregularity;
- a "dynamics shift" parameter that makes treated patients' pre-treatment transitions differ from untreated ones, deliberately breaking the time-homogeneity assumption.
The second part is the estimator. A CRONOS-style continuous-time forecaster is trained on all pre-decision transitions. A light predictive head reads the same encoded history and is fit to the forecast gap as in the objective above. It never sees post-decision images, which avoids conditioning on a treatment outcome.
The third part is an attribution analysis. We will measure how much of each model's predictive attribution falls on the velocity direction versus the true predictive direction. Attribution is computed with Expected Gradients over the image sequence, following Xiao et al..
Baselines
The naive reference is a static TARNet on the last pre-treatment scan, as in Xiao et al.. It tells us how much history matters at all. No reported numeric result was found on Lumiere or in the file for its static image benchmarks.
The first strong competitor is FlexTENet from Curth and van der Schaar with its encoder replaced by the same longitudinal sequence encoder we use. This is the natural "just add time" design. It shares prognostic structure, so it directly tests whether our separately trained forecaster adds anything. No result on longitudinal images is reported; the file says only that FlexTENet beat other learners across 101 simulation settings.
The second strong competitor is SyncTwin applied to frozen embeddings from the same image encoder. It also uses pre-treatment time as control information and has the most convincing real-data result in the file: −1.25 mmol/L estimated against a −1.26 mmol/L trial benchmark.
We also include a sequence T-learner, to show the false-positive failure mode in imaging, and delta features with a causal forest. The delta features are radiomics-style differences between pre-treatment scans, which reproduces the NSCLC pipeline with a control arm added.
The key ablation is our exact model with the forecaster trained only on untreated patients' outcomes, without the pre-treatment transition pool. This isolates the borrowed-time ingredient from the two-stage structure. A second ablation adds a representation-balancing penalty, to test the Crabbé et al. warning in the sequence setting.
For a fair comparison, all neural methods share the encoder, training data, compute budget, and the same number of hyperparameter trials. Hyperparameters are selected on a validation criterion available without counterfactuals: factual outcome error on held-out patients. Selecting on ground-truth PEHE would favor no method in a way a real user could reproduce.
Beating the field means two things. Our method must achieve lower PEHE and higher predictive attribution precision than longitudinal FlexTENet and embedding SyncTwin across most benchmark settings. The gap must also widen as the control arm shrinks and as the prognostic share of variance grows. A win that does not grow with those knobs would not support the mechanism we claim.
Why not just…?
Why not just plug a longitudinal encoder into FlexTENet or TARNet? This is the real threat. Shared-representation learners already absorb prognostic structure, and with a large control arm they may match us. Their prognostic fit is supervised only by decision-time outcomes, one label per patient. Our forecaster gets one label per visit pair, from every patient. The control-arm-size sweep settles the question. If FlexTENet matches us at every control fraction, the borrowed-time idea adds nothing, and we will report that.
Why not just use SyncTwin on image embeddings? SyncTwin's matching implicitly assumes treated histories lie in the span of control histories. It also estimates individual effects without exposing which image features modify them. If assignment depends on predictive features, treated histories may fall outside the controls' span. SyncTwin's own reliability bound, based on pre-treatment reconstruction error, should flag these cases, and we will report it alongside our results. Our method extrapolates through a learned dynamics model instead. That is better when dynamics are shared across patients and worse when they are not, which is exactly what the dynamics-shift knob measures.
Why not just compute delta features and fit a causal forest? Delta features computed after treatment are outcomes of treatment, not modifiers of its effect. Computed before treatment, they are mostly prognostic velocity. Svensson et al. and Crabbé et al. show that standard learners load such dominant signals into their effect estimates. We expect this baseline's velocity-attribution share to be highest of all methods. That share, not only PEHE, will show it.
Why not just carry forward the last image as the natural-history forecast? That assumes zero progression. It works for slow diseases but makes the forecast gap absorb all natural change. CRONOS beats this heuristic on Lumiere, 7.55 versus 8.38 NRMSE. We will run the carry-forward forecaster as a variant of our method so its cost is measured directly.
Proposed Experiments
On the synthetic OCT benchmark, we report PEHE, predictive attribution precision, velocity-attribution share, and the value of the treatment policy implied by each model's effect estimates. These are reported across the variance-ratio, control-fraction, confounding, and irregularity sweeps, with five generator seeds per setting.
The first decisive check is the transition-pool ablation. Removing the pre-treatment transitions should erase most of our advantage over FlexTENet. If it does not, our gains come from something other than the mechanism we propose.
The second decisive check is the dynamics-shift sweep. Our advantage should shrink and then reverse as treated and untreated pre-treatment dynamics diverge. A held-out-untreated forecast test should detect the divergence before effect estimates degrade, giving users a practical warning.
On real data, Lumiere tests only the forecaster, since it has no treatment arm. Our natural-history model must beat the last-context-image heuristic and approach the 7.55 NRMSE that CRONOS reports. If it cannot, the residuals it hands the effect model contain unexplained natural history, and the method cannot be trusted on real cohorts.
We do not claim to validate predictive biomarkers on the single-arm NSCLC cohort. Its lack of controls is exactly the limitation this work addresses. The real-data contribution is a forecaster that a future two-arm imaging cohort could use directly.
References
- Enhancing Predictive Imaging Biomarker Discovery Through Treatment Effect Analysis
- SyncTwin: Treatment Effect Estimation with Longitudinal Outcomes
- Temporal radiomics for non-invasive preoperative prediction of pathologic complete response to neoadjuvant chemoimmunotherapy in non-small cell lung cancer
- On Inductive Biases for Heterogeneous Treatment Effect Estimation
- Overview and Practical Recommendations on Using Shapley Values for Identifying Predictive Biomarkers via CATE Modeling
- Benchmarking Heterogeneous Treatment Effect Models through the Lens of Interpretability
- Really Doing Great at Estimating CATE? A Critical Look at ML Benchmarking Practices in Treatment Effect Estimation
- CRONOS: Continuous time reconstruction for 4D medical longitudinal series
- Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
- Longitudinal Causal Image Synthesis
- A Framework for Evaluating Predictive Models Using Synthetic Image Covariates and Longitudinal Data