Most clinical prediction labels are questions about time. Will this patient be readmitted within 30 days? Will they receive a new hypertension diagnosis within a year? Will they be transferred to the ICU in the next day? Each asks for the probability that a particular kind of event happens before a deadline. A model trained on unlabeled patient records to answer exactly that question, for every event type and every deadline, could answer these labels with almost no annotation. You would read the answer off the model and use the few labels only to correct its calibration. Current pretrained clinical models do not do this. They are trained to predict the next medical code, and a downstream classifier is then fit on their internal representations. The main obstacle is numerical integration. With a neural model of the event rate over time, the probability that an event occurs within one year is one minus the exponential of that rate integrated over the whole year. Computing that integral takes many network evaluations per patient, per task, per horizon, or else an error-prone simulation of the future one event at a time.
We propose to remove that obstacle. We pretrain a network that outputs the accumulated risk (the integrated rate, called the cumulative hazard) directly. The network is built so this quantity can only increase with time, following the construction of Fully Neural Network based Model for General Temporal Point Processes. The network is trained on unlabeled records to predict, for each event type, how long until that event type next occurs. Horizon-based labels then become a single forward pass, and few-shot learning shrinks to fitting a handful of calibration parameters. The claim is that when pretraining optimizes the same quantity the label measures, few-shot transfer on horizon tasks improves, especially on long-horizon diagnosis tasks where pretrained models currently lose to simple count-based methods.
Core Claim
A few terms first. A temporal point process is a model of events at irregular continuous times. Its intensity $\lambda(t)$ is the instantaneous event rate. Its cumulative hazard $\Lambda(t)=\int_0^t \lambda(s)\,ds$ determines the probability that no event has occurred by time $t$, which is $\exp(-\Lambda(t))$.
We parameterize a mark-conditioned cumulative hazard $\Lambda_\theta(\tau \mid m, h)$. Here $h$ is a backbone encoding of the patient's history up to an anchor time, $m$ is an event type (a code or ontology concept), and $\tau$ is time elapsed since the anchor. Monotonicity in $\tau$ is enforced with non-negative weights and monotone activations, as in the Integral Estimation Module of Addressing Mark Imbalance in Integration-free Marked Temporal Point Processes. The rate is recovered by differentiation, so no quadrature is ever needed.
The self-supervised target is the time from the anchor to the next occurrence of $m$, which we call the first-hitting time. It is censored at the end of the record when $m$ never recurs, and it is computed for many sampled anchors and marks per patient. The pretraining loss is the exact censored survival likelihood:
$$\mathcal{L} = -\sum_{(a,m)} \Big[\delta_{a,m}\log \partial_\tau\Lambda_\theta(\tau_{a,m}\mid m,h_a) - \Lambda_\theta(\tau_{a,m}\mid m,h_a)\Big].$$
In this loss, $a$ indexes anchors, $\tau_{a,m}$ is the observed first-hitting time (or the censoring time), and $\delta_{a,m}$ is 1 if the event was observed and 0 if it was censored.
A downstream task defined as "mark set $M$ occurs within horizon $H$" is answered with $\hat p = 1-\exp(-\Lambda_\theta(H\mid M,h))$. Few-shot adaptation fits a logistic calibration $\sigma(\alpha\log\Lambda_\theta(H\mid M,h)+\beta)$, optionally also tuning the embedding of $M$.
This differs from next-event modeling in a way that matters. A next-event model such as FlexTPP (Transformers for Mixed-type Event Sequences) gives an exact density over the time to the next event of any type. Asking "hypertension within a year" still requires marginalizing over every intervening event, which means sampling a rollout. EasyTPP reports that such rollouts accumulate errors over long horizons. Our head skips the intervening events and models each mark's first-hitting time directly. Integration-free parameterization is what makes this affordable. Every sampled (anchor, mark) pair costs one pass through a small network, not dozens of intensity evaluations across a year-long interval.
Relation to Prior Work
Three lines of work motivate the gap. EHRSHOT releases CLMBR-T-base, a model pretrained on next-code prediction, along with a few-shot benchmark. It finds that pretraining wins when labels are scarce. It also finds that count-based gradient-boosted trees catch up or win on one-year new-diagnosis tasks once labels grow past about 64 positives. We read that as evidence that next-code pretraining does not encode long-horizon occurrence well, and our objective targets exactly that. Pretext Training Algorithms for Event Sequence Data combines masked reconstruction of time intervals, contrastive views and real-versus-corrupted discrimination. Pretraining on 25% of the data beats training from scratch on all of it. However, its pretext tasks are unrelated to the downstream label, and a separate predictor must still be trained. Event Stream GPT frames generative pretraining as healthcare's universal task. Because it answers horizon questions only through generation, it inherits the rollout problem.
Two papers supply the method. The monotone cumulative-hazard network comes from Omi et al.. The mark-conditioned integral and the mark-first ordering come from IFNMTPP. IFNMTPP showed that conditioning time on mark stabilizes predictions for rare marks. Both papers train only supervised and only for the next event. We keep their parameterization but change the target to per-mark first-hitting times and train at pretraining scale on unlabeled data.
FlexTPP constrains the design. It shows that tokenizing continuous values loses precision, which argues against discretized time. Its continuous-valued lab handling is also the natural way to define derived marks such as "lab X abnormal." We do not claim to beat FlexTPP at next-event likelihood, and we do not attempt to.
Survival-style pretraining with binned (piecewise-constant) hazards is an obvious nearby design. It is treated below as a central ablation rather than dismissed. We could not verify from our sources whether such a clinical model already exists, so the novelty claim rests on the integration-free continuous head and on the calibration-only few-shot readout.
Research Program
The backbone is the released CLMBR-T-base. Initializing from it isolates the effect of the objective, because the backbone and its 2.57M-patient pretraining are held fixed. We attach the monotone mark-conditioned hazard head and run continued pretraining on the unlabeled records of EHRSHOT's training patients using the first-hitting loss. Marks include raw codes, ontology ancestors (so that composite task definitions such as "any hypertension code" have a learned embedding), and derived abnormal-lab marks. Anchors are sampled at visit boundaries, and marks are sampled with frequency-tempered weights so rare diagnoses are seen. A second, from-scratch variant pretrains the same architecture on EHRSHOT records alone, to check that the gains do not depend on CLMBR initialization.
Not every EHRSHOT task is an occurrence-within-horizon question. New diagnoses, readmission, ICU transfer and abnormal-lab tasks fit the survival readout directly. Long length of stay and the chest X-ray findings tasks are conditional on a future encounter, so they do not fit, and for them we fall back to a linear probe on $h$. We report the two groups separately rather than claiming uniform coverage.
We also measure the cost of integration directly. We use the same backbone with an intensity head trained under trapezoidal quadrature at 8, 32 and 128 nodes per interval, and record wall-clock time per training step, inference time per task query, and downstream accuracy. As a secondary, non-clinical check, we build horizon tasks on EasyTPP datasets, using StackOverflow badge and Retweet data through the EasyTemporalPointProcess codebase. An example task is "user earns badge type $m$ within $H$," with few labeled users, to test whether the effect holds outside medicine.
Baselines
The naive reference is EHRSHOT's count-based gradient-boosted trees with ontology expansion. This is the method that caught up with or beat the foundation model at larger $k$ on long-horizon diagnoses. It will be trained on the same $k$-shot splits, with the benchmark's hyperparameter search. Beating it means higher AUROC and AUPRC at every $k$ on the new-diagnosis tasks, including $k=128$ where it currently wins or ties. For EHRSHOT we found no per-task numeric result in our sources for either CLMBR-T-base or the trees, so we will reproduce both from the released code and weights and report our reproduced numbers alongside theirs.
The strongest existing method is CLMBR-T-base with EHRSHOT's standard linear-probe adaptation. It is the pretrained model built for this exact few-shot protocol and shares our backbone. We run it on identical splits and seeds. To remove any advantage from extra compute, we also run CLMBR given the same number of continued-pretraining steps on EHRSHOT training records under its original next-code loss. The central comparison is our survival readout with calibration against this compute-matched next-code model with a probe.
We include FlexTPP, fine-tuned from the released FlexTPP code on the same records. We query it two ways: a probe on its representation, and a Monte Carlo rollout estimate of the horizon probability with a sample budget chosen to match our inference time. This baseline tests whether exact next-event density plus rollout is as good as direct first-hitting modeling.
The ablation that isolates the key ingredient replaces the continuous monotone head with a piecewise-constant hazard over fixed bins of 1 day, 30 days and 1 year, trained on the same first-hitting targets. A further ablation keeps our pretrained model but discards the readout and probes $h$. It separates whether gains come from the objective shaping the representation or from label and model output being the same quantity.
On the EasyTPP horizon tasks, the supervised comparators are IFNMTPP and the pretext-trained model of Wang et al. For reference, IFNMTPP reports Macro-F1 0.1382 on USearthquake next-mark prediction. That is a different task, and it tells us only that its mark-first head is strong on rare marks.
Why not just…?
Why not just linear-probe a pretrained next-code model, as EHRSHOT does? This is the incumbent, and EHRSHOT's own finding that trees catch up on one-year diagnoses is the evidence that it underuses long-horizon structure. The compute-matched CLMBR comparison settles the question. If the probe equals our readout on the new-diagnosis tasks at all $k$, the claim fails.
Why not just bin time and use a discrete-time hazard? This is a real threat. For year-scale tasks a coarse binned head may match the continuous one, and we expect it might. The continuous head should matter for tasks whose horizons fall between or below bin edges, such as next-day ICU transfer or 30-day readmission, and for querying arbitrary horizons without retraining. The binned ablation evaluated at off-grid horizons will show whether that advantage is real. If it is not, we will report that the survival-query objective is the contribution and the integration-free head is a convenience.
Why not just roll out a generative model like FlexTPP or Event Stream GPT to estimate the probability? Rollouts are costly per query, and EasyTPP reports that they accumulate errors over long horizons. The matched-inference-time rollout baseline measures both effects directly.
Why not keep an expressive intensity head and integrate numerically? Omi et al. and EasyTPP report worse fit and higher cost from approximate integration. First-hitting targets over year-long gaps make this worse, because each loss term requires integrating over the entire gap. The quadrature sweep will show the accuracy-versus-cost curve. The proposal is justified only if matching our accuracy requires many more network evaluations.
Proposed Experiments
The decisive checks are four. First, the zero-shot readout, with no labels at all, must beat chance and the count-based trees at $k \le 8$ on the new-diagnosis, readmission, ICU-transfer and abnormal-lab tasks. This shows that the pretrained quantity really is the label. Second, at $k=64$ and $k=128$ on one-year new-diagnosis tasks, the calibrated readout must match or beat the trees, closing the gap EHRSHOT documents. Third, the calibration fit with two parameters should rival a full probe. We expect it to have lower Brier score, because the head is trained by likelihood and should already be close to calibrated. Fourth, the quadrature and binning sweeps should show that the continuous integration-free head reaches its accuracy at a fraction of the training and query cost.
All results are reported per task and per $k$, with AUROC, AUPRC and Brier score over the benchmark's resampled splits. Confidence intervals come from bootstrap over patients. The EasyTPP horizon tasks serve as a replication outside medicine. A null result on the first two checks would mean that matching the pretext objective to the label does not help, and we would report it as such.
References
- Fully Neural Network based Model for General Temporal Point Processes
- Addressing Mark Imbalance in Integration-free Marked Temporal Point Processes
- Transformers for Mixed-type Event Sequences
- EasyTPP: Towards Open Benchmarking Temporal Point Processes
- EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models
- Pretext Training Algorithms for Event Sequence Data
- Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events