Event-Effect Embeddings for Clinical Time Series

Hospitals record a patient's vital signs and lab values hour by hour, and many of the most important moments in that record are events: a patient is intubated, starts vasopressors, or begins dialysis. Self-supervised learning, which trains an encoder on unlabeled data by inventing prediction tasks from the data itself, has become the standard way to get useful patient representations without expensive labels. Current methods treat these events in one of two ways. Either they treat the event as noise that the representation should ignore, or they ignore it entirely and assume that nearby moments look alike. Clinical researchers ask a different question. In "temporal stratification," they split a timeline into before and after phases and compare each against a baseline of what would have happened anyway. Their goal is to measure what changed.

We propose a pretraining objective that borrows that logic directly. The encoder splits each patient-window embedding into two parts. The first is a stable part that should look the same before and after an event. The second is a changing part whose before-to-after difference serves as an "effect vector." The effect vector is trained against a counterfactual baseline: the drift the same patient would show over an equal stretch of time with no event. The claim is that this effect vector, the representation-level analogue of a difference-in-differences estimate, carries outcome-relevant information that existing event-based pretraining actively destroys. The same vector also gives a per-patient, label-free view of how an intervention changed a trajectory.

Core Claim and Relation to Prior Work

The closest prior work is Event-Based Contrastive Learning for Medical Time Series (EBCL). EBCL treats the windows just before and just after a recorded clinical event, from the same patient, as a positive pair. It then trains a Transformer with a CLIP-style loss to pull them together, using other patients' windows as negatives. This motivates our proposal, because it shows that events are useful anchors for pretraining on MIMIC-IV and a large heart-failure cohort. It also constrains our design. By making pre and post embeddings invariant, EBCL removes exactly the information that stratified clinical studies set out to estimate.

Those studies supply the logic we borrow. Excess burden of respiratory and abdominal conditions following COVID-19 infections compares infected patients against reweighted controls within each variant period. Cross-national comparison of viral transmission compares an intervention window against multi-year historical averages. In both, the quantity of interest is a deviation from a baseline, not a raw before-versus-after difference. We keep EBCL's invariance, but only on the stable part of the embedding. On the changing part we add a baseline-corrected effect term.

A second group of methods handles non-stationarity, but only to protect a smoothness assumption. Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding (TNC) uses a stationarity test to decide which neighboring windows count as positives. StatioCL uses the same test to build hard negatives across stationary and non-stationary segments. Soft Contrastive Learning for Time Series (SoftCLT) replaces binary pairs with weights that decay with time gap and raw-signal distance. All three treat a regime change as something that corrupts pair labels. None represents the change itself.

Phase-driven Generalizable Representation Learning for Nonstationary Time Series Classification (PhASER) goes further, deliberately cancelling phase drift for domain invariance. It also depends on dense waveforms, so it cannot be applied to sparse, irregular EHR data. Our proposal differs from this whole group in two ways. It targets irregular clinical records, and it treats the event-induced change as the signal rather than the nuisance.

Method

Let $x^-$ and $x^+$ be the windows of length $L$ immediately before and after an event of type $k$ for one patient. Let $c^-, c^+$ be a control pair from the same patient: two windows of the same length and gap, with no recorded event of that type between them. The encoder outputs $f(x) = (s(x), d(x))$, a stable part and a changing part. The stable part is trained with an EBCL-style contrastive loss on $(s(x^-), s(x^+))$. A small adversarial classifier that tries to tell pre-windows from post-windows using $s$ alone keeps event information out of $s$.

The changing part is trained through a drift model $g$, which predicts the event-free change from the pre-window and the gap. It is fit on control pairs by regressing $d(c^+) - d(c^-)$ on $g(d(c^-), \Delta)$, where $\Delta$ is the gap. The effect vector is then

$$e = \big(d(x^+) - d(x^-)\big) - g\big(d(x^-), \Delta\big),$$

which is the observed change minus the change expected without the event. This mirrors the "intervention window minus historical baseline" step of temporal stratification.

The effect vector is made informative by a contrastive identification task. Given $e$, the event type $k$, and $s(x^-)$, the model must pick the true $d(x^+)$ out of a batch of other patients' post-event windows of the same event type. Using the same event type prevents the model from solving the task by recognizing event-type artifacts. A decorrelation penalty between $s$ and $d$ keeps all patient identity from leaking into $d$.

Events are treatment initiations recorded in MIMIC-III and MIMIC-IV: intubation, vasopressor start, and renal replacement start. Hospital and ICU admission are also used, matching EBCL. Irregular measurements are handled with the event-sequence encoders in PyTorch-Lifestream.

Research Program

The work has three parts. First, we build the objective and reimplement EBCL in the same codebase, so the only difference is the loss. Second, we test whether effect vectors recover known effects. We plant shifts of controlled size and direction into a semi-synthetic version of the TNC four-state Gaussian-process simulator, adding interventions that change the state-transition rates at known times. We also inject synthetic level shifts into selected MIMIC-III channels after real event timestamps. The measure is the correlation between recovered and planted effects, plus whether effects too small to matter are correctly scored near zero.

Third, we test whether effect vectors add outcome information. We measure the improvement in linear-probe performance on post-event outcomes when $e$ is appended to the pre-event representation. We also check whether averaging $e$ within strata reproduces the direction of population-level differences estimated by a classical propensity-reweighted analysis of the same cohort. The strata are event types, and calendar periods split at the documented 2008 change in MIMIC-III's bedside charting system, a natural temporal break.

Baselines

The naive reference point is supervised training with no pretraining. Specifically, we use the multitask channel-wise LSTM of Multitask Learning and Benchmarking with Clinical Time Series Data. That paper reports an in-hospital mortality AUC-ROC of 0.870 on MIMIC-III (logistic regression: 0.848), a decompensation AUC-ROC of 0.911, a macro AUC-ROC of 0.776 on 25-phenotype classification, and a length-of-stay weighted kappa of 0.451. We rerun it on our splits, so our numbers are compared against our reproduction rather than the reported figures. At full labels we do not expect a large win over this baseline. Beating it means a higher score at 1% and 10% of labels, where TS-TCC showed that pretraining matters most on HAR (about 70% versus under 50% at 1% labels).

The strongest competitor is EBCL, because it already uses the same events as anchors and is the method our objective modifies. For EBCL on MIMIC-IV, the file says only that its AUCs were higher than those of other methods; no reported numerical result was found on Lacuna for EBCL. We will therefore reproduce it under the same conditions as our method: identical encoder, windows, event set, number of pretraining steps, and hyperparameter search budget, which is the same number of trials for each method. Beating EBCL means a better linear-probe AUC on post-event outcomes, averaged over five seeds, with confidence intervals that do not overlap. It also means at least equal performance on the four standard benchmark tasks, so that the effect term is not bought at the cost of general quality.

SoftCLT is the strongest general temporal contrastive method (+2.0% and +3.9% average accuracy over TS2Vec on UCR and UEA). We run it on hourly-gridded imputed versions of the records as a non-event reference.

A forecasting-residual baseline is included because it is the most direct simple competitor. A sequence model trained only on event-free stretches forecasts the post-event window, and the forecast error serves as the effect signal.

Ablations isolate the key ingredient. One removes the drift correction, setting $g = 0$, so effects become raw before-after differences. One removes the stable/changing split. A third takes EBCL's own post-minus-pre embedding difference as the effect vector.

Why not just…?

Why not just take EBCL embeddings and subtract pre from post? This is the most direct test of whether the new objective is needed, and it is run as an ablation. EBCL's loss explicitly shrinks that difference toward zero, so we expect it to carry little outcome information beyond the pre-window. If it matches our effect vector on the incremental-AUC test, the core claim is wrong, and we will report that.

Why not just detect breaks with a change-point test and run TNC or SoftCLT within segments? This repairs the smoothness assumption but still produces representations of states, not of changes. In our setting the event times are already recorded, so discovering breaks is not the bottleneck. TNC's authors also note that CPU-bound stationarity testing slows training. The forecasting and SoftCLT baselines bound how much simple segmentation helps.

Why not just forecast the counterfactual trajectory and use the residual? This is a serious threat, because it implements the same baseline-subtraction logic in data space. The difference is that raw-space residuals on sparse, irregularly measured variables are dominated by missingness and measurement noise, while our residual lives in a learned space trained to be identifiable. The forecasting-residual baseline settles this directly. If it matches us on effect recovery and incremental AUC, the simpler method should be preferred.

Why not just run a stratified propensity-weighted analysis, as clinical studies do? For population-average effects within prespecified strata, that remains the reference standard, as in the RECOVER analysis. We use it as an external check rather than a competitor. It cannot provide per-patient effect estimates or label-efficient downstream features, which are the two things this proposal adds.

Decisive Checks

Three results would settle the question. First, on semi-synthetic data with planted shifts, effect vectors should track the planted effect size. The no-drift-correction ablation should fail specifically when the planted effect is set to zero but background drift is present. Second, on MIMIC-III and MIMIC-IV, appending $e$ to pre-event representations should raise post-event outcome AUC over EBCL. This gain should exceed both the EBCL-difference ablation and the forecasting residual, at 1%, 10%, and 100% labels. Third, the average effect vector for each stratum should agree in sign with the classical reweighted estimates for the same event types. If the first holds but the second does not, the method is a measurement tool rather than a pretraining improvement. If neither holds, event invariance as in EBCL is the better default.