Many time-series models must decide before a sequence is finished. An intensive-care model may have to flag mortality risk from the first hours of a stay, and a sensor classifier may have to label an event from its first half. A useful trick is to train the early model, the "student", to mimic a "teacher" model that saw the whole sequence. The best current version of this trick, Generative Diffusion Prior Distillation (GDPD), handles the fact that one prefix can end in many ways. It trains a diffusion model, a generator that learns by gradually denoising random noise into samples, to produce plausible full-sequence teacher features given a prefix. That works, but it means training and repeatedly sampling a second generative model just to manufacture supervision targets.
Our observation is that when training data is synthetic, this second model is unnecessary. A procedural generator such as CauKer, which builds series by sampling Gaussian-process curves and passing them through random causal graphs, already knows the hidden ingredients behind each series. We can therefore fix a series' prefix and re-sample only its future from the generator itself. That gives K exact, cheap alternative continuations, each passed through a full-sequence teacher. The student learns from its prefix to predict the whole spread of those teacher features, not a single target. We pretrain one compact early-classification model this way on synthetic data and test whether it matches GDPD on real benchmarks while removing the diffusion prior's training and sampling cost. As a by-product, the predicted spread tells the model how much the unseen future could still change its answer.
Core Claim
For prefix-restricted time-series models, the distribution over full-context teacher features that GDPD learns with a diffusion model can be obtained almost for free. Branching the procedural generator that produced the training data supplies it. A student pretrained to match this branched distribution on CauKer-style data, then fine-tuned on real prefixes, should reach early-classification accuracy comparable to per-dataset GDPD at a fraction of the cost of generating supervision targets. Its predicted feature variance should also act as a better stopping signal than fixed waiting rules.
The idea connects the two directions in a specific way. Cross-temporal distillation supplies the mechanism: a privileged teacher that sees the future supervises a student that does not. Synthetic-data overhead is the constraint we attack, and it is also the lever we use. The expensive part of GDPD is a generative model built only to fabricate supervision. If the data is already synthetic, the data generator can take that role.
Relation to Prior Work
Three lines of work motivate the premise that future-aware teachers help prefix students. TimeKD has a language-model teacher read past and true future and distills its features and attention maps into a history-only forecaster. Di-Long distills a long-observation trajectory teacher into a short-observation student. Privileged history distillation for mammography distills multi-year screening histories into a student that runs when prior exams are missing. All three use the single future that the dataset happens to contain. Our method instead supervises with many futures consistent with the prefix.
GDPD supplies the method we build on and the strongest competitor. It already argues that a single static target is misleading and that sampled targets help. We keep that insight and replace its learned diffusion prior with exact sampling from the data generator. Our method is therefore restricted to settings where synthetic pretraining is possible, a restriction GDPD does not have.
CauKer supplies the data source. It shows that 100,000 procedurally generated series can pretrain an 8M-parameter Mantis model to 78.55% UCR accuracy, against 78.66% for 1.89 million real series. CauKer only trains full-sequence models and never uses its generator's knowledge of the future as privileged information.
Context-Matched Distillation constrains the design and is the main apparent contradiction. It shows that a bidirectional video teacher uses future camera turns to explain current frames, and that a causal student cannot imitate this at runtime. We read that failure as a teacher committing to one particular future the student cannot know. Branching addresses exactly this: averaged over K continuations, the target keeps only what the prefix actually determines, while the predicted spread records what it does not. We also stay in a non-generative, representation-learning regime, where TimeKD and GDPD report gains rather than harm. We do not claim the result transfers to autoregressive rollouts.
ProactiveLLM uses confidence signals such as entropy drops to decide when a streaming model has seen enough. We borrow that goal for our stopping test. Our sufficiency signal comes from branched futures rather than from the model's own confidence, which ProactiveLLM itself notes can fire too early on garden-path inputs.
Research Program
The generator stage reimplements the CauKer pipeline and adds one operation, branching. For each base series we record its latent description: the kernel compositions and hyperparameters, the causal graph, the activation functions, and the root curves. We choose a cut point $\tau$ from the earliness levels 20%, 50% and 80%. For the root Gaussian-process curves, we draw K futures from the exact Gaussian posterior given the root values up to $\tau$. For series of a few hundred steps this is a single Cholesky solve per root. We then push each branch through the same causal graph.
Branched futures are exact only where the graph's operations depend on the current or past time steps. We will audit the CauKer graph operations and modify any that look ahead in time. Because the generator also fixes hidden quantities the student never sees, the branched spread is narrower than the true ambiguity given only the observed prefix. We treat how much of the latent description is held fixed as an explicit knob, which we call branch depth. Holding everything fixed resamples only root-curve futures. Shallower branching also resamples kernel hyperparameters, or even the graph, conditioned on the observed prefix through cheap rejection sampling.
The teacher is a Mantis encoder pretrained on full-length CauKer series, which reproduces CauKer's setup. It runs once on each of the K completed branches, producing features $h^{(k)}$ for $k = 1, \dots, K$.
The student is a Mantis-sized encoder that reads only $x_{1:\tau}$, the series up to the cut point. It outputs a mean vector $\mu$ and a per-dimension variance $\sigma^2$, and is trained by the average Gaussian negative log-likelihood of the teacher features:
$$ \mathcal{L} = \frac{1}{K}\sum_{k=1}^{K} \sum_{d} \left[ \frac{(h^{(k)}_d - \mu_d)^2}{2\sigma_d^2} + \tfrac{1}{2}\log \sigma_d^2 \right], $$
where $d$ indexes feature dimensions. Under a squared-error loss alone, a single branch per series converges in expectation to the same conditional mean as K branches, so branching would only reduce variance. The variance term is what makes the K branches carry information that one sample cannot. We also test an energy-distance variant that matches the full set of branch features without assuming a Gaussian.
For real data, the pretrained student is fine-tuned on each real dataset's prefixes with an ordinary classification loss and no teacher. The comparison with GDPD therefore asks whether synthetic branched pretraining can replace per-dataset diffusion-prior distillation.
For cost accounting, we log GPU-hours for every stage: series generation, branching, teacher forward passes, and student training. For GDPD we log diffusion-prior training plus hint sampling. The file reports no measured training overhead for GDPD. It only describes the overhead as multi-step sampling latency, so we will measure it ourselves with the authors' configuration.
Baselines
The naive reference point is a prefix student pretrained on the same CauKer series by randomly cropping them to prefixes, with no teacher, then fine-tuned on real prefixes exactly as our method is. It shows how much of any gain comes from synthetic pretraining alone. Alongside it we report GDPD's own non-distilled student. GDPD reports 71.60% average AUC-PRC for that student on 12 UCR datasets at 50% prefix.
The strongest existing method is GDPD. On the same 12 UCR datasets at 50% prefix, it reports an average AUC-PRC of 84.64%, against 78.14% for standard logit distillation, according to GDPD. It is the strongest competitor because it explicitly models the many possible futures of a prefix, which is precisely our question. We run it with its published architecture on the same dataset subset and earliness levels. We give it the same student backbone where its design allows, and the same hyperparameter-search budget as our method. Beating it means matching its AUC-PRC within its seed-to-seed spread at 20%, 50% and 80% earliness while using clearly fewer GPU-hours for target generation, or exceeding it at equal cost. We also run GDPD on top of our pretrained student, to check whether the two methods are complementary.
Two ablations isolate the key ingredient. The first is single-future feature distillation on synthetic data: the student matches the teacher's features on the one original completion of each series. To equalize compute, it gets K times as many base series, so the total number of teacher forward passes is identical. If this matches our method, branching itself adds nothing beyond more data. The second is branched distillation trained with squared error only, which drops the variance term. Comparing it with the full method separates the benefit of a better mean target from the benefit of modeling the spread. Branch depth is swept as a third ablation.
On the forecasting side, TimeKD's reported gains of up to 9.11% in mean squared error on the ETT-family benchmarks give a reference point. We will cite it but not compete with it directly, because our main study is classification.
Why not just…?
Why not just pretrain on more synthetic data with prefix cropping? CauKer shows that synthetic data is cheap and effective, so a reviewer may argue that volume beats clever targets. TDScaling found that quantity scaling saturates early for agent trajectories, but that is a different domain. The relevant evidence is GDPD's 13-point margin over its plain student, which suggests prefixes need future-aware supervision. We settle the question directly by comparing against cropping-only pretraining at 1×, 4× and 16× data.
Why not just distill from the single real or synthetic future? This is the most serious threat. With enough data, single-future squared-error distillation estimates the same conditional mean as our method. Against it stands GDPD's reported gap between static-target logit distillation (78.14%) and sampled hints (84.64%). That gap was measured on real data with a learned prior, so it may not carry over. The compute-matched single-future ablation decides the matter. If it ties our method on accuracy and on stopping quality, we will report that branching is not needed and that the useful contribution is synthetic pretraining of prefix students.
Why not just keep GDPD's diffusion prior and make sampling faster? Few-step distillation of the prior, as discussed in work like Context-Matched Distillation, would cut sampling cost. It would still require training a prior for each dataset, and its samples would only approximate a distribution the generator already provides exactly. We include GDPD at its native step count and at a reduced step count, so readers can see where a sped-up prior lands on the cost–accuracy curve.
Why not just wait a fixed amount of time before deciding? ProactiveLLM shows that fixed waits such as Wait-3 and Wait-9 give a worse trade-off between accuracy and context consumed than adaptive triggers. We test whether the student's predicted variance gives a still better stopping rule than entropy-based confidence.
Proposed Experiments
The main experiment evaluates on the 12 UCR datasets used by GDPD, on the 12 UEA multivariate datasets, and on PhysioNet mortality, at 20%, 50% and 80% earliness, with five seeds each. We report AUC-PRC as GDPD does, alongside target-generation GPU-hours.
The decisive check has three parts, all at equal teacher-forward budget. Full branched distillation must beat single-future distillation. The variance-aware loss must beat squared-error-only branching. Our method must land within seed noise of GDPD's 84.64% at 50% prefix, or above it. If the first comparison fails, the central claim fails.
A second check tests whether branch spread measures real ambiguity. On held-out synthetic series, we compare the predicted variance against the true spread of teacher features under shallow branching. On real data, we use the variance as a signal for when to stop reading. We then plot accuracy against the fraction of each series consumed before deciding, and compare that curve with fixed waits and with entropy thresholds. We expect our signal to delay decisions specifically on series whose early segments are shared across classes.
A third check guards the full-sequence regime. It verifies that branched pretraining does not reduce full-length UCR accuracy below CauKer's 78.55%, which ensures we are not trading away ordinary performance for earliness.
References
- Generative Diffusion Prior Distillation for Long-Context Knowledge Transfer
- CauKer: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Data
- Efficient Multivariate Time Series Forecasting via Calibrated Language Models with Privileged Knowledge Distillation
- Distilling Knowledge for Short-to-Long Term Trajectory Prediction
- Longitudinal Risk Prediction in Mammography with Privileged History Distillation
- Context-Matched Distillation: Teacher Causality for Autoregressive Video Distillation
- ProactiveLLM: Learning Active Interaction for Streaming Large Language Models
- Beyond Quantity: Trajectory Diversity Scaling for Code Agents