Attention models for irregular time series, such as ICU vitals recorded at uneven intervals or clickstreams with real elapsed times, turn each timestamp into a vector using a bank of learned sine and cosine frequencies. How well these models work depends heavily on which frequencies they use. An ensemble of models with different frequency banks, some tuned to short bursts and some to slow trends, is the obvious fix, but it multiplies serving cost by the number of members. The standard cheap substitute is weight averaging: average the members' parameters and deploy a single model. That substitute should break exactly here. Averaging two sinusoid frequencies gives a third frequency that neither member used, so models with different time banks are not interchangeable in weight space.
This note proposes a single change that avoids the problem. Average every parameter that weight averaging normally handles well, but do not average the frequency banks. Instead, stack the members' frequency banks side by side and stack the first linear maps that read them. For the first attention layer, this merge reproduces the ensemble's average queries, keys and values exactly. Everything downstream is averaged in the usual way. The deployed model is one network whose time embedding is $M$ times wider, where $M$ is the number of members. That embedding is a small fraction of total compute, so inference stays effectively single-pass while keeping time-scale diversity that plain weight averaging would erase.
Core Claim
In a continuous-time attention layer, the timestamp of an event enters through a functional embedding $\phi(t)$ and is added to the event's content embedding $x$. The sum then passes through linear projections such as the query map $W_Q$. Suppose member $i$ has its own frequency bank, giving embedding $\phi_i$, and its own projection $W_Q^i$. Then the members' average query is
$$\frac{1}{M}\sum_{i=1}^{M} W_Q^i\big(x+\phi_i(t)\big) \;=\; \bar W_Q\, x \;+\; \tfrac{1}{M}\big[W_Q^1,\dots,W_Q^M\big]\begin{bmatrix}\phi_1(t)\\ \vdots \\ \phi_M(t)\end{bmatrix},$$
where $\bar W_Q$ is the plain average of the members' query maps. The time term on the right is exactly a single projection applied to the stacked frequency banks. The same identity holds for keys and values. The merged model therefore matches the ensemble's average projections in the first layer exactly. Downstream layers are averaged as in Diverse Weight Averaging for Out-of-Distribution Generalization (DiWA), which trains models from a shared starting point with mildly varied hyperparameters and averages their weights.
The claim is that this "stack-the-clock" merge keeps most of the accuracy and calibration gain of an $M$-model output ensemble over diverse time scales, at roughly single-model latency. It should also clearly beat three cheaper options: naive weight averaging of the frequencies, DiWA with a shared frequency bank, and a single model trained from scratch with an equally wide frequency bank. The last of these is the real threat, and the experiments are built to settle it.
Grounding in Prior Work
The temporal side supplies the architectures and the motivating weakness.
Self-attention with Functional Time Representation Learning builds Bochner embeddings (sampled sine/cosine frequencies) and Mercer embeddings (a truncated Fourier basis with learned coefficients) for self-attention over event sequences. It reports that Mercer embeddings are sensitive to the truncation size $k$ and that Bochner embeddings have high variance. That is the motivating evidence that no single frequency bank is safe.
Multi-Time Attention Networks for Irregularly Sampled Time Series (mTAN) embeds time with periodic and linear terms and attends from fixed reference times to observations. It reaches 0.858 AUC on PhysioNet mortality while training far faster than ODE-based models, but it commits to one embedding and one reference grid. Attend and Diagnose (SAnD) shows that masked attention can replace recurrence on MIMIC-III, at the cost of task-specific, hand-tuned look-back windows. Both papers supply the backbones we reuse. Neither considers ensembling or merging.
The ensemble side supplies the merging method and its known limits. Averaging Weights Leads to Wider Optima and Better Generalization (SWA) averages checkpoints from one training trajectory. DiWA averages independent runs and argues, through its decomposition of error into bias, variance, covariance and locality, that diversity across runs lowers the covariance term that single-trajectory averaging leaves behind. Both methods also constrain our design: they work only when the members sit in a shared low-loss basin. Members with different frequencies are the case most likely to violate that condition.
Learning Neural Network Subspaces trains a line or simplex of models and evaluates its midpoint. Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation (LatentBE) averages rank-one subnetwork factors into a single weight matrix. Both are vision-only and average every parameter. Our method differs in one respect: it identifies the parameter group that must not be averaged, and replaces averaging there with an exact linear merge. Collaborative Learning for Deep Neural Networks trains several heads on a shared trunk and keeps one. We include it only as a cheap alternative, not as a method we reuse.
Learning Pattern-Specific Experts for Time Series Forecasting Under Patch-level Distribution Shift handles temporal regime changes by routing input patches to separate expert networks at test time. That keeps several experts live during deployment, which is the inference cost this proposal tries to avoid.
Research Program
The first step is a diagnostic. We take mTAN and a Bochner/Mercer self-attention model, and train pairs of members from a shared warm start. Some pairs share a frequency bank and some have deliberately different banks, for example frequencies initialized at short, medium and long time scales. We then measure validation loss along the straight path between paired members under two merges: naive averaging, and the stacked-clock merge. The prediction is a clear loss barrier under naive averaging only when banks differ, and a small barrier under stacking. If even same-bank attention members show large barriers, DiWA-style averaging fails for temporal attention in general. That is a reportable negative result, and it would push the method toward the subspace-training variant described below.
The second step is the method itself. We train $M$ members, with $M$ between 4 and 8, from a shared warm start. Each member gets its own frequency bank, and learning rate and dropout are mildly varied as in DiWA. We merge them by stacking the time banks and their first projections and averaging everything else.
A training-time variant follows the subspace idea. During joint training we sample mixing weights over members, evaluate the merged model directly, and so push the downstream weights toward compatibility. This costs one training run rather than $M$.
We also test how far the exact merge can be pushed. In mTAN the time embedding feeds the reference-time queries as well. We check whether stacking banks inside the reference-to-observation attention keeps the gain, or whether the softmax nonlinearity erodes it.
Baselines
The naive reference point is a single mTAN or single functional-time attention model, trained with its default frequency bank and with tuning matched to the method's budget. On PhysioNet 2012 mortality, mTAN reports 0.858 AUC (Multi-Time Attention Networks). On MIMIC-III, SAnD-Multi reports 0.908 AUROC on decompensation and 0.771 macro AUC on phenotyping (Attend and Diagnose). On Stack Overflow, Mercer attention reports 46.83% accuracy; on Walmart.com it reports 14.94% Hit@10 against 12.49% for discrete positional encoding (Self-attention with Functional Time Representation Learning). We rerun all of these ourselves rather than relying on copied numbers.
The accuracy ceiling is the output ensemble of the same $M$ diverse-bank members, run at $M$ times the inference cost. We found no reported result on Lacuna for an ensemble of mTAN or functional-time attention models. Our own run of this ensemble is therefore the target, and the method succeeds if it closes most of the gap between the single model and the ensemble.
The strongest zero-overhead competitor is DiWA applied to temporal attention: $M$ members sharing one frequency bank, with all weights averaged. It uses the same number of members, the same training compute and the same hyperparameter perturbation ranges. We found no reported result on Lacuna for DiWA on temporal models. The closest reported figure is DiWA's 68.0% against SWAD's 66.9% on DomainBed (Diverse Weight Averaging). That benchmark is image classification, so it indicates that cross-run diversity can help but is not comparable to our tasks.
We also run SWA on a single trajectory and standard knowledge distillation from the ensemble into one model, each with matched training compute. Two ablations isolate the key ingredient. The first averages frequencies naively instead of stacking them. The second trains a single model from scratch with the same $M$-times-wide frequency bank. Beating DiWA and the wide-bank model on AUROC and on negative log-likelihood, with confidence intervals over at least five seeds, counts as success.
Why not just…?
Why not just use SWA? Single-trajectory checkpoints share one frequency bank and make correlated errors. DiWA's analysis attributes SWAD's shortfall on DomainBed to exactly this unreduced covariance. SWA is nonetheless cheap and may suffice on dense, regular data. We report it everywhere, and if it matches our method under sparse observation, the method is not needed.
Why not just distill the ensemble into one model? LatentBE reports that standard distillation keeps only the ensemble's mean prediction and loses calibration: on CIFAR-100, its negative log-likelihood was 1.038 against 0.792 for LatentBE. That evidence comes from images. We test distillation directly on the clinical tasks with matched compute.
Why not just train one model with a wider frequency bank? This is the serious threat. The merged model has exactly this architecture, so any advantage must come from how the parameters were found, not from capacity. The evidence on our side is weak but real: Mercer embeddings are known to overfit as $k$ grows. We would argue that independently trained members spread their frequencies more usefully than joint training does. The wide-bank ablation, run with a tuned $k$ and weight decay, settles the question. If it ties our method, the honest conclusion is that time-scale ensembles reduce to wider embeddings, and we would publish that finding.
Why not route inputs to time-scale experts, as TFPS does for patches? Routing keeps several experts live at test time and adds routing logic. Our target is one fixed set of parameters at deployment.
Decisive Experiments
The decisive tests are run under time-scale shift, where diversity across time scales should matter most. We train on PhysioNet 2012 and MIMIC-III at full observation density, then test with 50% to 90% of observations dropped, following mTAN's sparsity protocol. We also test on held-out MIMIC-III care units.
For each method we report AUROC, negative log-likelihood, expected calibration error and wall-clock latency per patient. The claim fails in any of three cases: the stacked merge does not beat both DiWA and the wide-bank model under the sparsest condition, the linear-path barrier analysis shows no difference between stacking and naive averaging, or the merged model recovers less than half of the gap between the single model and the ensemble. Stack Overflow and Walmart.com serve as event-sequence replications.
References
- Diverse Weight Averaging for Out-of-Distribution Generalization
- Self-attention with Functional Time Representation Learning
- Multi-Time Attention Networks for Irregularly Sampled Time Series
- Attend and Diagnose: Clinical Time Series Analysis Using Attention Models
- Averaging Weights Leads to Wider Optima and Better Generalization
- Learning Neural Network Subspaces
- Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation
- Collaborative Learning for Deep Neural Networks
- Learning Pattern-Specific Experts for Time Series Forecasting Under Patch-level Distribution Shift