Hindsight Distillation for Streaming Irregular Time Series

Models that read a sparse, irregularly timed record, such as an ICU chart where a lab value appears every few hours and heart rate every few minutes, work best offline. There they can look both backward and forward in time and fill each gap with a smooth estimate. A bedside alert cannot wait for the future. This proposal trains a strictly online model, one that only sees the past, to predict what an offline model would have concluded at the same moment if it could see the whole record. The online model also predicts how uncertain that guess is, and the uncertainty should grow as the time since the last measurement grows. At deployment the online model never touches future data. It has simply learned, during training, which parts of hindsight are predictable from the past and which are not.

The reason this matters specifically for sparse data is the training signal. A common way to give an online model "future awareness" is to make it forecast upcoming raw observations. In sparse records those observations exist only at a few random times and in a few channels, so most of that signal is missing. An offline teacher instead supplies a dense, denoised target at every query time on a continuous grid, including inside long empty gaps. Our central prediction is that the gain from distilling the teacher's hindsight, compared with forecasting raw future observations, should widen as the fraction of missing observations grows.

Core Claim

A causal continuous-time student trained to regress a non-causal teacher's latent state, with a learned variance that discounts unpredictable hindsight, will close a measurable part of the offline–online gap on irregular clinical and sensor data. It does this without leaking future information. The advantage over raw future-forecasting auxiliaries should grow with sparsity.

Where This Sits in Prior Work

The motivating evidence comes from On the Choice of Interpolation Scheme for Neural CDEs and its earlier version Neural Controlled Differential Equations for Online Prediction Tasks. A neural CDE is a network whose hidden state evolves continuously, driven by an interpolated path through the observations. These papers show that the standard natural-cubic-spline path is non-causal. On a synthetic Brownian-motion task, a spline-based neural CDE scores 81.7%, above the 75.0% that any causal model can reach. They then build causal paths: rectilinear control, and cubic Hermite splines with backward differences. These papers also give us the student architecture. They do not ask whether the information the offline model exploits could be transferred to a causal model during training, and we do ask that.

Two papers supply teachers that smooth bidirectionally. Multi-Time Attention Networks (mTAN) attends from fixed reference times to all observations, past and future. ContiFormer puts ODE-defined keys and values inside attention over the full sequence. We use both as offline teachers, and a causally masked mTAN as a second student family.

The transfer idea comes from video. PreFM predicts "pseudo-future" features on uniform frames and distills from a frozen foundation model. It reports that unconstrained pseudo-future features hurt in chaotic scenes, and that failure constrains our design. Our student predicts the teacher's state at the present time, not future features, and a learned variance down-weights cases where hindsight cannot be inferred from the past. EDICT shows that uncertainty in continuous-time models should expand over observation-free intervals. We use that as a checkable prediction about our student's variance, not as a component. TURTLE shows that strictly causal history models can rival bidirectional ones in video restoration. It assumes uniform frames, so it is evidence that the gap is closable rather than a method we reuse.

Research Program

We first train a teacher offline on the full record. The teacher is either a natural-spline neural CDE or a bidirectional mTAN, and it produces a latent $z_T(t)$ at any query time $t$. We then train a causal student, a Hermite-backward-difference neural CDE, which outputs a mean $\mu_S(t)$ and a diagonal variance $\sigma_S^2(t)$ using only observations at times $\le t$. The training loss is

$$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \sum_{t \in Q} \left[ \frac{\lVert z_T(t) - \mu_S(t) \rVert^2}{2\sigma_S^2(t)} + \tfrac{1}{2}\log \sigma_S^2(t) \right].$$

Here $\mathcal{L}_{\text{task}}$ is the online prediction loss evaluated at every query time, $\lambda$ is a weight, and $Q$ is a dense grid of query times plus all observation times. Because $Q$ includes gap interiors, the student gets supervision where raw data offer none. The prediction head reads $\mu_S(t)$ and $\sigma_S(t)$ alongside the student's own state, in the spirit of PreFM's fusion of predicted context with current features.

The analysis has three parts. We measure how the gain depends on time since the last observation. We test whether $\sigma_S$ grows across gaps, as EDICT's results suggest it should. We measure where in a sequence the student agrees with the teacher, near onsets and inside long gaps. Causality holds by construction because the student never reads future inputs, and we verify it empirically.

Baselines

The naive reference is a causal GRU on hourly bins with forward-fill. It shows what a deployment team would do without continuous-time machinery.

The strongest existing causal method is the online neural CDE of Morrill et al. (paper). On MIMIC-IV it reports a length-of-stay RMSE of 0.099, against 0.148 for GRU-D and 0.154 for ODE-RNN, and a mortality AUC of 0.943. We rerun it with our preprocessing, splits, and hyperparameter search budget, using the same student architecture, so that the only difference is the distillation term. Beating it means a statistically reliable improvement across five seeds on the rerun numbers. We will also report against the published figures, while noting that preprocessing may differ.

For discrete EHR modeling we include TPC in a causal configuration. It reports a remaining length-of-stay MAD of 2.28 days on MIMIC-IV.

Offline teachers serve as upper references, not competitors. mTAN reports an AUC of 0.858 on PhysioNet 2012 mortality and an accuracy of 0.910 on Human Activity. For the UEA archive, ContiFormer reports only average ranks: no reported per-dataset accuracy found on Lacuna for ContiFormer on UEA.

Three ablations isolate the key ingredient. The first, a self-distillation control, uses a causal teacher of the same architecture; if it matches the full method, the gain is regularization, not hindsight. The second, a raw-forecasting auxiliary, has the student predict the next observed values and their arrival time, with an equal tuning budget. The third, a fixed-variance version, removes $\sigma_S$ to test whether discounting unpredictable hindsight matters.

Why not just…?

Why not just bin, forward-fill, and use a fast causal network? The evidence in the file argues against it. Forward-fill biases physiological trends (He et al.) and erases the signal carried by when tests are ordered (Albers et al.). ContiFormer's lead over discrete models also widens at 70% missingness. We keep this baseline anyway, because a well-tuned binned model is cheap and occasionally strong.

Why not just add an auxiliary loss that forecasts future observations? This is the real threat. It gives future awareness with no teacher. Our claim is specifically that teacher targets are denser and less noisy than sparse raw targets. The sparsity sweep on UEA (30/50/70% dropped) and on further subsampled PhysioNet settles it. If raw forecasting matches hindsight distillation at all sparsity levels, the teacher is unnecessary and we will say so.

Why not just accept a small fixed delay and run the offline smoother? In some applications this is acceptable, so we treat it as a reference curve rather than dismissing it. We will plot accuracy against wall-clock delay $\tau$ for the teacher run on a delayed window, with the student as the $\tau = 0$ point. A delay counted in observations is unbounded in wall-clock time when the next lab may be hours away. A delay counted in wall-clock time is well defined, and the curve shows how many hours of waiting the student is worth.

Why not just use standard knowledge distillation on the teacher's output probabilities? Soft labels carry hindsight only at the decision layer and only at labeled times. We include logit-only distillation as an ablation to test whether dense latent targets add anything beyond it.

Proposed Experiments and Decisive Checks

The datasets are MIMIC-IV (online mortality, length of stay, sepsis, scored at every hour), PhysioNet 2012 (mortality and online interpolation), Human Activity, and 20 UEA datasets with 30/50/70% of observations dropped. We report AUROC and AUPRC, RMSE or MAD, sepsis lead time, per-update latency, and the number of solver function evaluations.

The first decisive check is the Brownian-motion test. The teacher should score near 81.7%. The student must stay at or below 75.0%, and ideally near it. Exceeding 75% would reveal a leak in our pipeline. Approaching 75% from the non-distilled student's score would show that distillation transfers legitimate information rather than cheating.

The second check is the interaction between missingness and the method. Hindsight distillation's margin over the raw-forecasting auxiliary should grow monotonically from 30% to 70% missingness.

The third check is calibration. $\sigma_S$ should increase with time since the last observation, and down-weighting by $\sigma_S$ should help most on the noisiest channels. That would directly address PreFM's hallucination failure.

If the self-distillation control explains most of the gain, the hindsight claim fails, and we will report the method as regularization.