Continuous Time at No Inference Cost: Mergeable Adaptation of Frozen Time-Series Models to Irregular, Data-Scarce Domains

Many real time series do not arrive on a regular clock. Hospital measurements, cluster traces and field sensors produce observations at irregular moments, and a target domain often supplies only a few dozen such series for training. Large pre-trained time-series models such as MOMENT assume uniformly spaced patches, so they are poorly suited to this data. The usual fixes create a different problem. One option is to make the small trained add-on, a low-rank adapter (LoRA, which learns a weight change $\Delta W = BA$ with thin matrices $B, A$), depend on the time gap. A gap-dependent weight change differs for every token, so it can no longer be folded into the frozen weights. The adapter must then run as a separate branch at inference time, and unfused low-rank branches have been measured to cost up to 30% extra latency (Sparse High Rank Adapters).

This proposal argues that time-dependence does not need to live in the weights. Every effect of irregular timing that matters under data scarcity can be placed in two spots where an inference engine already pays the cost. The first is the patch-embedding matrix multiply, which can absorb gap features as extra input columns. The second is the elementwise rotation of queries and keys that fused attention kernels already perform for rotary position encoding. If continuous elapsed time drives that rotation, attention becomes a function of real time differences. Everything else is a static LoRA that is merged into the weights after training. The resulting deployed model has the same compute graph as the base model. The open empirical question is whether this restricted design gives up accuracy relative to per-token time-conditioned adapters. We expect it does not when target data is scarce, and the experiments can show otherwise.

Core Claim

Adapting a frozen, regularly patched time-series model to irregularly sampled, low-resource target domains can be done with zero added inference kernels. The method combines three pieces. A merged static LoRA absorbs the domain shift. A low-rank gap embedding is folded into the patch projection. A learned continuous-time rotary phase is applied to a subset of attention head dimensions. The claim has two parts. Accuracy should match or beat time-conditioned weight modulation of the Terra or CoLoRA kind at 10 to 50 target series. Measured latency should be indistinguishable from the unadapted base model. If the restricted design loses clearly to per-token weight modulation, the result is still useful, because it locates the data scale at which paying the latency becomes worthwhile.

Why This Is Not Already Done

The closest work splits along the latency axis. Two papers make low-rank updates depend on a continuous variable, and both pay for it at inference:

  • Time-Varying LoRA (Terra) writes $\Delta W(t) = W_{\text{up}} K(t) W_{\text{down}}$. Its $t$ is a synthetic index interpolating between image domains, not physical time. It is used offline to generate augmentation data, and its three-matrix branch cannot be merged per token.
  • CoLoRA generates time-dependent low-rank scalings with a hypernetwork for PDE surrogates, trained from as few as 10 trajectories. That hypernetwork must run before each step, and the approach is built for coordinate networks rather than transformers.

The zero-latency side has no notion of time:

The time-series PEFT work assumes regular grids. TRACE factorizes MOMENT's forecasting head. C-LoRA and ST-LoRA condition adapters on channel or node identity. PETSA adapts at test time with backpropagation. Continuous-time models such as ContiFormer handle irregularity well, but they are trained from scratch and their attention is quadratic and costly.

None of this work asks which time effects can be moved into operations that are already fused, or whether doing so costs accuracy. That question is what this proposal studies.

Research Program

Tokenizing irregular series. The irregular target series is split into patches of a fixed number of consecutive observations, not fixed wall-clock windows. The frozen model therefore still sees a regular token sequence. Each patch $i$ carries two pieces of time information:

  • its continuous timestamp $\tau_i$, the first observation time normalized by a domain time scale;
  • a gap vector $g_i$, the within-patch inter-observation gaps together with the missingness mask.

Folding gap features into the embedding. The gap vector enters through a low-rank map that is folded into the existing patch embedding. Let $x_i$ be the patch values, $W_e$ the frozen embedding matrix, $\phi$ a fixed set of sinusoidal and log-gap features, and $U, V$ thin trainable matrices with $V$ initialized to zero:

$$ e_i = W_e x_i + U V \phi(g_i) = \begin{bmatrix} W_e & UV \end{bmatrix} \begin{bmatrix} x_i \\ \phi(g_i) \end{bmatrix}. $$

After training, the product $UV$ is precomputed. The embedding then becomes a single, slightly wider matrix multiply, and no new kernel is added.

Continuous-time rotary phase. Queries and keys in a chosen subset of head dimensions are rotated by a block-diagonal rotation $R(\omega\tau)$. The frequencies $\omega$ are learned and initialized at zero, so training starts exactly at the pre-trained model. This is the same safety property that LoRA gets from zero-initializing $B$. Because rotations compose, the attention score between patches $i$ and $j$ depends only on the real elapsed time between them:

$$ \langle R(\omega\tau_i) q_i,\; R(\omega\tau_j) k_j \rangle = \langle q_i,\; R(\omega(\tau_j - \tau_i))\, k_j \rangle . $$

The sign follows the convention that the rotation's transpose equals its inverse. This gives a relative-gap attention signal of the kind built explicitly with pairwise interval matrices in Exploring Periodicity and Interactivity in Multi-Interest Framework for Sequential Recommendation. It does not need the $O(L^2)$ matrix materialization that the dossier flags as a cost of those methods. Its only runtime cost is computing cosines and sines of real-valued $\tau$ instead of integer positions.

Static LoRA for the domain shift. A static LoRA on the attention and feed-forward projections absorbs the domain shift. Low rank also acts as a regularizer: it has been reported to limit overfitting to spurious artifacts (Parameter-Efficient Fine-Tuning with Low-Rank Adaptation for Vision Transformers) and to preserve calibration (On the Calibration of Low-Rank Adaptation for Large Language Models). The LoRA is merged at deployment, HiWi-style.

Analysis of expressivity. We will characterize what the restricted design can express. It can produce additive gap effects on the input, and multiplicative recency effects through attention. It cannot change a projection's weights per token depending on $\Delta t$. The study then measures how much that missing capacity matters as the number of target series grows. The hypothesis has two parts:

  • at small data, per-token weight modulation overfits;
  • the crossover point moves later as the missingness rate rises, because the gap embedding carries more of the signal at higher missingness.

Baselines

Do-nothing reference. Frozen MOMENT receives irregular series placed onto its grid using its native mask, with only a linear probe trained on top. MOMENT reports linear probing as competitive with specialized models such as PatchTST on regular data. We found no reported result on Lacuna for MOMENT on irregular UEA or Time-IMM data. This baseline marks the floor and also measures the base latency that every other method is compared against.

Interpolate-then-merge. Irregular series are resampled onto a uniform grid by linear interpolation, with the mask kept as an input. A merged static LoRA is then trained with the same rank and step budget as our method. This baseline also has zero latency, so it is the most direct threat to the proposal.

Strongest existing irregular-time method: ContiFormer. It is trained from scratch under the protocol of ContiFormer, which drops 30%, 50% or 70% of observations on 20 UEA datasets. ContiFormer reports the best average rank among all its baselines, between 1.9 and 2.4. The file gives no per-dataset accuracies, so we will rerun it ourselves on our low-data splits, with matched tuning trials.

Forecasting reference: Time-IMM pipeline. On Time-IMM we compare against the timestamp-to-text and gated multimodal fusion pipeline of Time-IMM. That pipeline reports a 6.71% average MSE reduction from adding text, and 38.38% on ClusterTrace. We use only its numeric-irregularity components, unless we also add text to our model.

Strongest time-conditioned adapter competitors. We test two:

  • a Terra-style per-token adapter $B K(\Delta t) A$, in its linear and exponential forms;
  • a CoLoRA-style hypernetwork that emits a per-token low-rank scaling from $\tau$.

Both are run unmerged, with trainable parameter counts matched to our method within 10%. All methods get the same target data, the same 24-trial hyperparameter search, and the same number of optimizer steps.

Full fine-tuning. This is reported where memory allows.

Ablations. Three ablations isolate the ingredients. The first is static LoRA plus gap embedding, with no rotary phase. The second is static LoRA plus rotary phase, with no gap embedding. The third replaces our rotary phase with an additive Time2Vec-style embedding of $\tau$, which isolates the benefit of relative-time attention over absolute time features.

What counts as beating a baseline. Against the Terra-style and CoLoRA-style adapters, we need equal or better accuracy at 10 and 50 target series, together with lower measured latency. Against interpolate-then-merge, we need a statistically significant gain at 50% and 70% missingness across five seeds. Against ContiFormer, we need a better average rank at 10 to 50 series per dataset.

Why not just…?

Why not just interpolate onto a regular grid and use ordinary merged LoRA? This is the real threat. It is equally fast and conceptually simpler. Interpolation invents values inside long gaps and erases the information that a gap exists, while the Time-IMM benchmark finds that irregularity itself is informative. ContiFormer also reports that its lead over discrete models widens significantly at 70% dropped observations. Neither result proves our method will win, though. The missingness sweep settles it. If interpolate-then-merge stays within noise of our method at 70% missingness, then the time encoding is unnecessary and we will report that.

Why not just accept the latency of a time-conditioned LoRA? The branch is small, and in some offline settings the cost may be acceptable. Against this, unfused LoRA has been measured at up to a 30% inference penalty, and per-token conditioning rules out the merge that removes it. The cost also grows for streaming workloads with many short requests. We will measure the actual overhead on GPU with fused attention and on CPU. We will also report the accuracy each variant buys per millisecond, so a reader can decide whether the trade is worth making.

Why not just train ContiFormer, a model built for continuous time? With 10 to 50 target series there is little data to learn temporal structure from scratch. The Universal Domain Adaptation Benchmark for Time Series finds that higher-capacity models collapse under domain shift. ContiFormer's continuous attention integrals also cost more than fused dot-product attention. The data-scale curve will show where, if anywhere, from-scratch continuous modeling overtakes adapting a pre-trained prior.

Why not just adapt at test time, as PETSA does? PETSA handles non-stationarity by running backward passes during inference. That is the latency cost this proposal aims to avoid, and PETSA assumes regular patches. Our method is complementary: a merged adapter could serve as PETSA's starting point. We will not claim to address drift within a single deployment.

Proposed Experiments

Classification. We use the 20 UEA datasets under ContiFormer's 30/50/70% drop protocol. Training sets are subsampled to 10, 50 and 200 series per dataset, plus the full set. The metric is accuracy and average rank over five seeds.

Forecasting. We use the nine Time-IMM datasets with training history truncated to match the same data regimes, measuring MSE.

Secondary cross-subject setting. HAR and HHAR are used from the UniDA benchmark in closed-set form, with irregular dropping applied. Here the source subjects are the pre-adaptation data and the scarce target subjects are the adaptation data.

Latency protocol. Every model is exported in its deployment form, with merges applied where possible. We measure median and p99 latency per batch at batch sizes 1 and 64, on an A100 using PyTorch scaled-dot-product attention with rotary encoding and on a desktop CPU. We also count kernel launches. Our method passes the latency test if it adds no kernels and its latency stays within 2% of frozen MOMENT.

Timestamp-permutation check. Within each test series we shuffle the timestamps while keeping the values fixed. A model that genuinely uses time should degrade. If our model does not degrade, the rotary and gap components are inert. This mirrors C-LoRA's channel-shuffling test.

Crossover analysis. We plot the accuracy gap between our method and the Terra-style adapter against the number of target series, separately for each missingness rate. The proposal's central claim holds if the gap is zero or favors our method up to at least 50 series. It fails if per-token modulation leads already at 10 series.