Clinical notes are full of time references that point at something other than the date the note was written: "two days after surgery," "on post-op day 3," "at 0600," "prior to admission." To turn these into calendar dates, a system has to work out which earlier moment each phrase counts from, and then do the date arithmetic correctly. Current clinical language models are pre-trained by predicting hidden words, which teaches them clinical vocabulary but never asks them to keep track of what date the story has reached. Normalization is then handed off to rule engines or prompted chatbots, and that handoff fails. In the 2024 chemotherapy-timeline shared task, scores fell by at least 0.2 F1 when systems had to find and resolve time expressions in raw notes instead of receiving them pre-annotated.
We propose a pre-training objective that teaches a clinical encoder to keep a "running clock." Hospitals hold millions of notes that contain explicit, easily parsed dates. We hide those dates and train the model to recover each one in two parts: it points to an earlier moment in the record to count from, and it predicts a signed offset in days. A small deterministic calendar program then computes the final date, so the network never has to write out digits. No human labels are needed. The anchor-plus-offset form is exactly what relative-expression normalization requires, and because only offsets are learned, the model is unaffected by the per-patient date shifting used to de-identify clinical data.
Core Claim
Continued pre-training of a clinical encoder with a self-supervised "where is the clock, and how far has it moved" objective will improve normalization of relative and incomplete clinical time expressions more than either of two cheaper routes: ordinary continued masked-word pre-training on the same notes, or supervised training of the same anchor-and-offset output layer on labeled data alone. The gain should be largest on expressions that count from moments other than the note date, which make up most clinical cases.
The case for this rests on three findings. Normalization of Relative and Incomplete Temporal Expressions in Clinical Narratives showed that clinical relative expressions anchor to admission (53%), a previous time mention (37%), or discharge (16%), whereas 96% of newswire expressions anchor to the creation date. It solved the problem with separate SVM classifiers for anchor and direction plus a rule parser, reaching 57.2% strict and 82.09% relaxed accuracy. Our output form, an anchor pointer plus an offset, is their decomposition. The difference is that we learn it from unlabeled text inside the language model, instead of from hand-built features. Prompting Large Language Models for Clinical Temporal Relation Extraction showed that the clinically pre-trained GatorTron-Base (345M parameters) beats LLaMA3-8B on i2b2 temporal relations, reaching 89.54% F1 with prompts that match its pre-training objective. It never normalized expressions to dates, and its pre-training was plain masked-word prediction. That paper supplies our starting checkpoint and an argument that the pre-training objective, not model size, is the lever. Finally, the Overview of the 2024 Shared Task on Chemotherapy Treatment Timeline Extraction released an unlabeled corpus of longitudinal notes for roughly 89,000 cancer patients intended for continued pre-training. Teams instead treated normalization as an external Timenorm or regular-expression step. That corpus is our training data.
Research Program
The pre-training signal comes from explicit dates. In each unlabeled note we find fully specified absolute dates with a high-precision rule normalizer. Timenorm, the grammar system from A Synchronous Context Free Grammar for Time Normalization, is suitable. We keep only unambiguous matches. For each such date at position $i$, we build a candidate anchor set $A_i$: the note's metadata date, admission and discharge dates when the record supplies them, and earlier absolute dates in the same note or in earlier notes from the same patient. Each anchor $a$ has a known date $t_a$. We then hide the date, along with every repeated mention of the same date, so the model cannot simply copy it from nearby. The model must recover it.
The model never says which anchor is "correct," because several anchors are consistent with any given date. The anchor is therefore treated as a hidden choice, and training maximizes the total probability over all consistent explanations:
$$\mathcal{L}_i = -\log \sum_{a \in A_i} p_\theta(a \mid x)\, p_\theta(\delta = d_i - t_a \mid a, x),$$
Here $x$ is the note with the date hidden, $d_i$ is the true hidden date, $p_\theta(a \mid x)$ is a pointer distribution over candidate anchors computed from the encoder state at position $i$, and $p_\theta(\delta \mid a, x)$ is a distribution over signed day offsets. The offsets are bucketed on a log scale with an exact-day sub-head. This loss pushes the model toward anchors from which the offset is easy to predict given the text, which is what a reader does.
We add a coarser variant in which only the month or year is recoverable, so the model also learns granularity. Ordinary masked-word prediction stays in the training mix. CHRONOBERG found that sequential continued training of Pythia models causes sharp forgetting, and that EWC, a regularizer that penalizes changes to important weights, reduces the forgetting but makes the model too rigid. Mixing objectives within a single training stage avoids that trade. Following BioBART, which found that sentence shuffling hurt biomedical models because discourse order matters, we never permute sentences, and we feed earlier same-patient notes as preceding context.
For downstream normalization, a relative expression such as "the following Tuesday" or "POD#2" is resolved with the same pointer head, which predicts the anchor. A small operator head predicts a calendar operation drawn from the primitive set used by ARTime, for example "move to next unit" or "shift by v units." The calendar program executes the operation. Because the network emits only pointers, operator labels, and offsets, month lengths and leap years are handled by code. This sidesteps both the digit-fragmentation problem of standard tokenizers and the arithmetic slips that generative models make.
We will also analyze what the model learns. We will probe whether the encoder's implied clock is consistent across a note, meaning whether successive positions move forward monotonically. We will also check whether anchor choices match the admission, previous-mention, and discharge distribution that Sun et al. annotated.
Baselines
The naive reference point is the rule pipeline used in the chemotherapy shared task: cTAKES for preprocessing plus CLUlab Timenorm, anchored to the note date. It requires no training, and its ceiling is set by the note-date assumption that clinical text violates.
The strongest published system on our main metric is the SVM anchor-and-direction pipeline of Sun et al. on i2b2 relative and incomplete expressions. It reports 57.2% strict normalization accuracy, 82.09% relaxed accuracy (within one day), and 74.68% anchor-point accuracy. We reuse their task definition and evaluation, so beating it means exceeding 57.2% strict accuracy and 74.68% anchor accuracy on the same split. We expect the largest margin on strict accuracy, since exact-day arithmetic is where pipeline errors accumulate.
The strongest modern competitor is a GatorTron-Base model with our exact anchor-and-operator output layer, fine-tuned only on i2b2 labels. This is the model that Prompting Large Language Models for Clinical Temporal Relation Extraction showed to be strongest for clinical temporal reasoning. It uses the same calendar program, the same fine-tuning budget, and the same hyperparameter search as our model. The only difference is the absence of clock pre-training.
The key ablation isolates the pre-training objective. We continue pre-training GatorTron-Base on the same chemotherapy-timeline notes, for the same number of tokens and the same compute, with masked-word prediction only, and then fine-tune it identically. If this ablation matches our model, the gain came from seeing clinical notes, not from the clock objective.
A prompted large-model baseline follows Discourse-Aware In-Context Learning for Temporal Expression Normalization. It uses target-centric exemplars with a preceding-sentence context window, which that paper reports as comparable to or better than supervised baselines on i2b2. No exact i2b2 figure is reported in the file, so we will rerun it. We will run it twice: once producing dates directly, and once producing anchor and operator choices for our calendar program. The prompt-tuning budget is matched to our fine-tuning budget.
A fifth baseline is synthetic relativization. We rewrite explicit dates in the unlabeled notes into templated relative phrases, then fine-tune the plain model on those pairs. This tests whether data augmentation alone captures the gain.
All encoder variants share the tokenizer, the checkpoint, and three random seeds. We report confidence intervals and count a result as a win only when it exceeds seed variance.
Why not just…?
Why not just keep improving rules or grammars? They do not hallucinate and need no GPUs. However, their failure is in choosing the anchor, not in doing arithmetic. Sun et al. show that the note-date assumption covers only a minority of clinical cases. The shared-task baseline built on Timenorm is part of the setting where scores dropped at least 0.2 F1 on raw text, and roughly half of ARTime's errors came from surface patterns it had never seen. We keep the deterministic part, the calendar program, and learn only the part rules get wrong.
Why not just prompt a large model with discourse context? This is a real threat, especially when paired with our calendar program. Gautam et al. nonetheless report that relative and under-specified expressions remained the main failure mode even with context windows. In the shared task, fine-tuned smaller models beat zero-shot large models. The prompted-plus-calendar baseline settles the question directly. If it matches our model on strict accuracy for admission-anchored and previous-mention-anchored expressions, the pre-training objective is not worth its cost, and we will report that result.
Why not just continue masked-word pre-training on the same notes? Standard domain-adaptive pre-training is cheap and often enough. It is also what GatorTron already does, and GatorTron has no notion of an anchor. Our equal-compute masked-word ablation is designed to answer this. Our prediction is that it helps with recognizing expressions but leaves anchor accuracy roughly unchanged.
Why not just fine-tune on labeled data, or on synthetic relative phrases? i2b2 has only 310 summaries, so supervised training sees few examples of the rare anchor types. Templated rewrites teach surface forms but not the implicit narrative clock, the "the next morning" kind of reasoning the hidden-date objective forces. Both are run as baselines. If synthetic relativization closes most of the gap, we will reframe the contribution as data generation rather than pre-training.
Proposed Experiments
The primary test is i2b2 relative and incomplete normalization, measured by strict and relaxed accuracy and broken down by gold anchor type. The decisive pattern is a gain over both the masked-word ablation and the supervised-only model that is concentrated on admission-anchored and previous-mention-anchored expressions.
The second test is end-to-end chemotherapy-timeline construction on raw notes (Subtask 2), using the official month-level F1. We also add a day-level re-scoring, because the official granularity hides arithmetic errors. The file reports only the gap of at least 0.2 F1 between gold-input and raw-text settings, so we will rerun the baseline and measure how much of that gap our model closes.
The third test checks robustness to date shifting. We re-shift every date in the MIMIC-style test notes by a random per-patient offset. Our model's accuracy should be essentially unchanged, while a direct date-generating model should drop. This matters given that EHRCon found over 56% of note-versus-table inconsistencies to be time-related.
The fourth test checks transfer and forgetting. We evaluate on TempEval-3 news text, where ManTIME reports 0.77 value accuracy and ARTime 75.4%. Clinical clock training should not hurt there, because the note-date anchor is simply one candidate. We also re-measure i2b2 temporal-relation F1 to confirm the model has not lost ground relative to the 89.54% GatorTron reference.
References
- Normalization of relative and incomplete temporal expressions in clinical narratives
- Prompting Large Language Models for Clinical Temporal Relation Extraction
- Overview of the 2024 Shared Task on Chemotherapy Treatment Timeline Extraction
- A Synchronous Context Free Grammar for Time Normalization
- Chronoberg: Capturing Language Evolution And Temporal Awareness In Foundation Models
- BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
- Automatic rule generation for time expression normalization
- Discourse-Aware In-Context Learning for Temporal Expression Normalization
- EHRCon: Dataset for Checking Consistency between Unstructured Notes and Structured Tables in Electronic Health Records