Specialist language models, such as clinical, biomedical or financial ones, are trained once and then frozen, but the language they serve keeps moving. New drugs appear, coding standards are revised, and guidelines change. A model trained on older text degrades on newer text roughly in proportion to how far the vocabulary and meaning have shifted. Retraining the whole model each year is expensive, and naive continued training on small batches of new text tends to erase what the model already knew. We propose to split a specialist model's adaptation into two parts. The first is a static adapter, a small set of added weights shared across all years, which holds what does not change, such as physiology or the discourse structure of a discharge summary. The second is a dynamic adapter whose weights are generated from a low-dimensional time state. That state moves smoothly from year to year and is deliberately kept small, so anything stable is pushed into the static part.
The bet is concrete. When a new period arrives with only a little text, updating a 64-dimensional time state plus a generator shared across periods should beat both retraining a monolithic adapter and fitting a fresh adapter per period. The model should also remain correct on older text, because the older time states are still there to query. This is the static-dynamic factor decomposition from recommender systems and author modeling, moved out of the input embedding and into the weights of a pre-trained Transformer.
Core Claim
The closest prior work is Deep dynamic neural networks for temporal language modeling in author communities. It conditions a small LSTM, trained from scratch, on a fixed author vector plus a yearly author vector. The yearly vector evolves by a residual update, $h_{a,t} = h_{a,t-1} + f_\phi(h_{a,t-1}, h_a)$, and the combined vector is fed in as the first token. That paper motivates our design: the decomposed model beat a time-token baseline on Semantic Scholar titles (micro perplexity 74.3 versus 77.2). It differs from ours in three ways. It decomposes a per-author input vector rather than domain-wide model weights. It never touches a pre-trained model. And it only models titles under 20 tokens.
We claim the same smooth-trajectory prior helps more when it governs the weights of a pre-trained specialist Transformer. We further claim the payoff is largest in the regime that D-CPT Law warns about. That paper fits loss scaling laws for continual pre-training and finds that at low domain-data ratios, adding data can raise domain loss. A new period's text is exactly such a small stream.
Concretely, for each adapted weight matrix $W_0$ of a frozen specialist model, the weight used at period $t$ is
$$W_t = W_0 + B_s A_s + \mathrm{LoRA}(G z_t),$$
where $B_s A_s$ is a low-rank static update shared across all periods. $z_t \in \mathbb{R}^k$ is a small time state, with $k$ around 64. $G$ is a shared linear generator that maps $z_t$ to the factors of a second low-rank update. The low-rank parameterization reuses the technique of Low-Rank Weight Decomposition for Efficient Model Adaptation, which adapts weights through $\Delta W = BA$ for a single static task.
Training minimizes the language-modeling loss on each period's text plus a trajectory penalty:
$$\lambda \sum_t \lVert z_t - z_{t-1} - f_\phi(z_{t-1}) \rVert^2 + \mu \sum_t \lVert z_t \rVert^2 .$$
The first term is the residual transition from Delasalles et al. and plays the role of the Kalman dynamics in Dynamic Poisson Factorization. The second term makes time-varying capacity costly, so content shared across periods migrates into the static update. Without it, the split is not identifiable. That is the failure mode, factors leaking into each other, that Disentanglement Beyond Static vs. Dynamic documents for sequential models.
Research Program
The application setting is biomedical and clinical text, where the drift is documented. Time Matters splits MIMIC-IV-Note, BioNER and BioASQ into four chronological intervals and shows two things. First, semantic drift correlates with downstream degradation, with Pearson r often above 0.85. Second, old models fail on emerging entities while newer models stay backward compatible. It diagnoses the problem and explicitly leaves the temporal-adaptation remedy open. We supply one.
We build the method on two backbones. One is an encoder, a BioRoBERTa-class model, to match the Time Matters protocol. The other is a decoder from the Pythia family, which is the base family that Efficient Continual Pre-training used for FinPythia. Any decoder scale beyond the roughly 1B range is conditional on compute.
The program has three parts. First, we fit the model on periods $T_1$ to $T_3$ and add $T_4$ as a streaming update. In that update, $W_0$, $B_sA_s$ and $G$ are frozen, and only $z_4$ is estimated, initialized from the transition prediction $z_3 + f_\phi(z_3)$. Second, we vary the amount of $T_4$ text from about 1% to 100% of a full period to trace data-efficiency curves. Third, we analyze what the dynamic part learned. We ask which tokens' likelihoods change most along the time-state trajectory, and whether they concentrate on new drug names, ICD-10 terms and COVID-era vocabulary rather than on general clinical language. That check tests whether the decomposition separates what it should, rather than just adding parameters.
Baselines
The naive reference point is the specialist model frozen at the end of $T_1$ to $T_3$ and evaluated unchanged on $T_4$. Time Matters shows this degrades. We report its drop as the floor any method must improve on.
The strongest practical competitor is sequential continual training of a single LoRA adapter on $T_4$ text, with replay of a fixed fraction of earlier-period text. It matches our trainable-parameter count and token budget, and its learning rate and replay ratio get the same tuning sweep. A reviewer would reach for this first, and it is likely to be competitive when $T_4$ data is plentiful.
The strongest published method for forgetting-aware domain adaptation is DAS from Continual Pre-training of Language Models. DAS soft-masks gradients on important units and adds a distillation-style regularizer and a contrastive loss. It reached 77.93 average Macro-F1 across six unlabeled domain corpora with a backward-transfer score of −1.09. Because that number is on a different benchmark (sequential distinct domains, RoBERTa-base), we rerun DAS on our temporal splits with the same backbone and budget rather than compare figures across papers. We are aware of no published result for any adaptation method on the Time Matters four-interval splits; the summarized results report only drift-degradation correlations, not per-task scores.
For the language-modeling sanity check on Semantic Scholar titles, the comparison is Delasalles et al.'s decomposed model, which reports prediction-task micro perplexity of 74.3. Beating it means lower perplexity with a pre-trained backbone under their temporal prediction split. We acknowledge that a pre-trained Transformer may win this through scale alone. So we also report the time-token and no-trajectory ablations on the same backbone.
Two ablations isolate the key ingredient. In the first, per-period adapters are fitted independently, with the same parameters but no generator and no transition penalty. In the second, the static update is removed and $\mu = 0$. Beating the baselines means two things. In the low-data regime (10% of a period or less), the method should be better on $T_4$ task scores, ICD-10 macro-F1 and protein NER F1, than tuned sequential LoRA with replay. And it should lose less on $T_1$ to $T_3$ than either sequential LoRA or DAS. If the gap vanishes at full data, we report that rather than claim a general win.
Why not just…?
Why not just keep training one adapter on the newest data, with replay? This is the real threat, and it may tie us when the new period is large. Our argument rests on the low-data regime: D-CPT Law finds a low-ratio inflection near 5% domain proportion where extra data hurts, and the "less is more" result in Efficient Continual Pre-training (top 10% of curated data at +12.45% versus all data at +8.3%) shows small, noisy streams are hard to exploit. Estimating 64 numbers is better posed than re-fitting thousands of adapter weights. The data-efficiency curve is designed to settle this. If sequential LoRA matches us at 1–10% of a period, the decomposition has no practical value and we will say so.
Why not just prepend a year token? Delasalles et al. tested a temporal-conditioning baseline that tied the static model (77.2 versus 74.3 perplexity on S2, 113.1 versus 110.8 on NYT). That evidence is from LSTMs, though, and attention layers may use a date token better. We include a year-token-plus-LoRA baseline on the same backbone. The decisive difference is that a year token cannot be extrapolated to a period whose token was never trained, whereas $z_t$ can be initialized from the transition.
Why not just retrieve recent documents at inference time? Retrieval handles new facts well, and we treat it as complementary rather than beaten. But EHRNoteQA shows accuracy drops as more notes enter the context. Time Matters implies that the embeddings used for retrieval drift too. And retrieval does not fix tasks like NER and coding, where the model must read new terminology inside the input. We run a retrieval-augmented variant on the drug-interaction subset and report whether it plus our adapters beats either alone.
Proposed Experiments
The primary experiment uses the Time Matters four-interval protocol on MIMIC-IV-Note top-50 ICD-10 prediction and BioNER. We use the same chronological splits and report $T_4$ forward performance, the $T_1$ to $T_3$ backward score, and the data-efficiency curve from 1% to 100% of $T_4$. We add full-label ICD-10 coding from Mimic-IV-ICD, whose reported macro-F1 of 5.42% on the full ICD-10 label set shows how hard rare codes are. There we test whether the dynamic factor helps codes that appear mainly in later years.
For the decoder, we evaluate on the emerging-drug pharmacology subset of ClinicBench (codebase), where nearly half of evaluated models scored worse than random. The sanity check is the S2 titles temporal prediction task against 74.3 micro perplexity.
The decisive checks are three. First, at 10% of $T_4$ data, the method should beat tuned sequential LoRA with replay on $T_4$ while losing less on earlier periods. Second, removing the transition penalty or the static update should erase most of that gain; otherwise the benefit is just capacity. Third, the tokens whose probability moves most along $z_t$ should be the new-terminology tokens that Time Matters identifies as drifting, not general vocabulary. If the first check fails, the proposal fails. If the second fails, the decomposition is not what is helping. If the third fails, the "static" and "dynamic" labels are not earned.
References
- Deep dynamic neural networks for temporal language modeling in author communities
- D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
- Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data
- Dynamic Poisson Factorization
- Disentanglement Beyond Static vs. Dynamic: A Benchmark and Evaluation Framework for Multi-Factor Sequential Representations
- Time Matters: Examine Temporal Effects on Biomedical Language Models
- Efficient Continual Pre-training for Building Domain Specific Large Language Models
- Continual Pre-training of Language Models
- EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries
- Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classification
- Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark