Many important time series arrive at uneven times. ICU vital signs, clinical visit logs and climate stations all have this property, and the gap between two observations is itself informative. Transformer-style models handle such data by attaching time stamps to tokens. They do not track how much they know about the present after a long silence. Kalman filters, which maintain an explicit error covariance (a matrix describing current uncertainty), handle this naturally. In practice, though, they break when trained by gradient descent in low-precision arithmetic, because the covariance loses positive definiteness.
We propose a sequence-mixing layer that combines the two. It is a drop-in replacement for attention whose memory is updated by an exact, time-aware Kalman filter, and it is implemented in square-root form. Square-root form means the layer stores a triangular factor of the uncertainty matrix rather than the matrix itself, and updates it with orthogonal rotations. The layer should therefore have three properties. It should forget at a rate set by real elapsed time. It should report a calibrated variance for every query. It should train stably in bfloat16 even when gaps range from seconds to days.
Core Claim
Gated KalmaNet (GKA) shows that a sequence layer can be read as online ridge regression from keys to values, derived from the Kalman filter. GKA keeps an explicit covariance, unlike Mamba-style layers that drop it. It reports over 10% relative gains over linear state-space baselines on RULER retrieval and long-question tasks, and up to 2x the throughput of Transformers at 64K tokens.
GKA buys its stability with two assumptions. It assumes a steady-state filter, and it uses an input-dependent regularization gate solved by Chebyshev iteration. Both assumptions fail under irregular sampling. After a long gap the prior uncertainty over the memory should inflate. After a dense burst of near-duplicate keys, the information matrix becomes badly conditioned. A steady-state approximation cannot represent either swing.
Our claim is that a transient, continuous-discrete Kalman filter over the layer's memory solves both problems. Its forgetting is driven by the actual interval $\Delta t_k$. It is propagated in square-root information form with QR updates, so its factor has condition number equal to the square root of the covariance's. Such a layer should match strong irregular-time attention models on accuracy, give uncertainty that grows with gap length, and remain finite in bfloat16 where covariance-form and steady-state variants diverge.
Concretely, each head treats its memory matrix $W$ as a latent state with an Ornstein–Uhlenbeck-style prior: between observations it relaxes toward zero with rate $\lambda$. Each token is an observation $v_k = W k_k + \text{noise}$, made at real time $t_k$. Let $\Lambda_k$ be the information matrix, the inverse covariance of $W$ shared across value dimensions. Let $\gamma_k = e^{-\lambda \Delta t_k}$ be the time-dependent retention factor, and let $\rho$ be the prior precision. The exact update is then
$$\Lambda_k = \gamma_k \Lambda_{k-1} + (1-\gamma_k)\rho I + k_k k_k^\top .$$
We never form $\Lambda_k$. Instead we keep its upper-triangular factor $R_k$, with $\Lambda_k = R_k^\top R_k$. We obtain it as the triangular part of a QR decomposition of a stacked matrix:
$$\begin{pmatrix} \sqrt{\gamma_k}\, R_{k-1} \\ \sqrt{(1-\gamma_k)\rho}\, I \\ k_k^\top \end{pmatrix} = Q \begin{pmatrix} R_k \\ 0 \end{pmatrix}.$$
From this factor the layer reads two quantities by triangular solves. The first is the output $W_k q_k$. The second is a predictive variance $\|R_k^{-\top} q_k\|^2$, which says how well the memory covers query $q_k$. When nothing has been observed for a long time, $\gamma_k \to 0$ and the variance returns to the prior $1/\rho$. When observations are dense, the variance shrinks.
Relation to Prior Work
Three lines of work each cover two of the three ingredients.
Irregular-time attention models supply the regime and the benchmarks but carry no uncertainty. Multi-Time Attention Networks (mTAN) attend from a fixed grid of reference times to irregular observations, training 85–100x faster per epoch than ODE models. ContiFormer lets keys and values evolve along linear ODEs and integrates attention over time. Both produce deterministic features and cost quadratic memory in sequence length. We replace their pairwise routing with a recurrent memory that is both uncertainty-aware and linear in cost.
Square-root filtering for irregular data motivates the stability choice. Neural Continuous-Discrete State Space Models (NCDSSM) put a Cholesky-factored continuous-discrete Kalman filter in a latent model and report that the covariance form diverged in training. Its filter, however, tracks a low-dimensional physical state, not a key–value memory, so it has no associative recall. Parallel square-root statistical linear regression shows that standard covariance filters produce NaN estimates in float32 on GPUs while square-root versions do not. This is the constraint that rules out covariance-form designs for us.
The method is supplied by Robust Automatic Differentiation of Square-Root Kalman Filters via Gramian Differentials. It gives finite gradients through the QR step even when the stacked matrix is wide or rank-deficient, but proves this only for linear Gaussian models. Our per-head filter is exactly such a model: keys act as observation matrices and $\gamma_k$ as a time-varying transition. The Gramian adjoint therefore applies directly. Our contribution there is to extend it to $\Delta t$-dependent transitions and to differentiate with respect to $\lambda$.
Kalman Bayesian Transformer applies Kalman smoothing only to a transformer's output layer and keeps a full covariance. We put the filter inside the token mixer instead.
Research Program
We will build the layer in JAX, reusing the open-source square-root filtering code released with the Gramian-differentials paper. Rate $\lambda$, observation noise and prior precision $\rho$ are learned per head. Keys and queries come from the usual projections plus a time embedding of absolute $t_k$, so the layer can still use periodic structure.
The cost concern is real. The rank-one key update costs $O(d^2)$ per token, but the identity "floor" term makes each step a full-rank $O(d^3)$ refactorization. We will therefore use a chunked scheme in the spirit of GKA. Within a chunk of $C$ tokens, outputs are computed from the factor at the start of the chunk plus a small $C \times C$ Cholesky-factored correction. One block QR at the chunk boundary absorbs all decay, floor and key terms. Applying the floor once per chunk, using the summed $\Delta t$, is an approximation. We will measure its error against the exact per-token recursion.
The analysis has three parts. First, we will bound the conditioning of $R_k$ as a function of $\rho$, $\lambda$ and the spread of $\Delta t$, explaining why a fixed regularization gate is miscalibrated under bursty sampling. Second, we will verify that the reported variance matches the true posterior variance on synthetic data generated from the assumed model. Third, we will test hybrids that interleave one softmax attention layer per few Kalman layers, mirroring GKA's finding that hybrids are needed for recall.
Baselines
Naive reference. A vanilla Transformer with continuous sinusoidal time embeddings of $t_k$ and $\Delta t_k$, the "just add time to the tokens" model. It uses the same depth, width and optimizer and receives the same hyperparameter search budget as our model, 30 trials per dataset. Beating it means lower interpolation MSE and higher classification accuracy on every benchmark below.
mTAN Multi-Time Attention Networks is the strongest attention baseline on the clinical tasks. It is fast and reports mortality AUC 0.858 on PhysioNet 2012 and accuracy 0.910 on Human Activity. We rerun its released configuration with our data splits and the same tuning budget, and also report its published numbers. Beating it means matching 0.858 AUC within the seed spread while additionally supplying calibrated variances. On interpolation MSE it should be beaten outright at 70–90% missingness.
ContiFormer ContiFormer is the strongest attention baseline on irregular classification. It reports a best average rank of 1.9–2.4 on the UEA archive subsampled at 30/50/70%. The file reports ranks rather than per-dataset accuracies, so we will rerun it ourselves. Beating it means a better average rank under identical subsampling masks, at lower memory on long series.
NCDSSM NCDSSM is the strongest square-root filtering baseline. It reports forecasting MSE 4.74 on CMU walking (NCDSSM-LL) and 0.34 on USHCN (NCDSSM-NL). We compare against these figures with their protocol. Beating or matching it means equal MSE with a non-Markovian, recall-capable layer.
S2P2 Deep Continuous-Time State-Space Models for Marked Event Sequences is the scale reference on MIMIC-II and EHRSHOT. It handles sequences of more than 500,000 events where Transformers run out of memory. No reported likelihood figure was found on Lacuna for EHRSHOT, so we will rerun it.
Ablations. Three ablations isolate the key ingredients:
- GKA unchanged, with time only in its token embeddings.
- GKA with $\Delta t$ fed into its forgetting gate, keeping the steady-state Chebyshev solver. This is the critical ablation.
- Our exact transient filter in covariance form with diagonal jitter and symmetrization in place of QR.
All three run at matched parameters in both float32 and bfloat16.
Why not just…?
Why not just feed $\Delta t$ into Gated KalmaNet's gate? This is the most serious threat, and we treat it as a co-equal hypothesis. It may well match us in float32 on mildly irregular data. Our argument for why it should fail elsewhere is that its steady-state solver and fixed regularization gate assume conditioning that does not swing. Irregular sampling makes conditioning swing by orders of magnitude between gaps and bursts. The second ablation, combined with a sweep over $\Delta t$ dispersion, settles the question. If it matches us on accuracy, divergence rate and calibration across that sweep, we will report that the contribution reduces to time-dependent forgetting and drop the square-root claim.
Why not just add jitter $\epsilon I$ and symmetrize a covariance-form filter? Jitter is added after the cancellation has already destroyed digits, as Duník et al. explain. The parallel square-root regression paper reports NaN divergence of covariance forms in float32. Still, our information-form update is additive, so cancellation is milder than in the classical update, and jitter might suffice in float32. The third ablation measures this directly, reporting NaN rate and gradient-norm blowups in bfloat16.
Why not just put a Gaussian output head on mTAN or ContiFormer? Such a head learns variance as a function of features. Nothing forces it to grow over unobserved intervals or to reflect how much data the model has seen near a query. We will test this by comparing variance against gap length and by scoring calibration (negative log-likelihood and coverage of 90% intervals) on held-out interpolation points. If the head calibrates as well, the uncertainty argument weakens, and the remaining case rests on stability and linear cost.
Proposed Experiments
Accuracy. We run PhysioNet 2012 interpolation at 50–90% observed and mortality AUC, Human Activity classification, UEA classification with ContiFormer's subsampling, and USHCN and CMU walking forecasting. Each uses the original protocols, five seeds, and the baselines above.
Scale. We train on MIMIC-II and EHRSHOT event streams and record memory and wall-clock against S2P2 and mTAN, up to the 500,000-event lengths S2P2 reports.
Decisive check 1: stability. On a synthetic sampler where we control $\Delta t$ dispersion from regular to heavy-tailed bursts, we train every Kalman-family variant in bfloat16. We record divergence rate, gradient finiteness and the condition number of the stored factor.
Decisive check 2: calibration. On PhysioNet interpolation, the predictive variance should rise monotonically with distance to the nearest observation and give near-nominal interval coverage, where the time-embedded Transformer and the Gaussian-head mTAN do not.
Decisive check 3: recall. On a synthetic irregular-time associative recall task, adapted from RULER-style retrieval with random timestamps, we test whether the pure Kalman stack needs one softmax layer to match attention. This checks whether GKA's hybrid finding carries over to irregular time.
The proposal is falsified if the time-gated GKA ablation matches the square-root layer in bfloat16 across the dispersion sweep, and the Gaussian-head mTAN matches its calibration.
References
- Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regression
- Multi-Time Attention Networks for Irregularly Sampled Time Series
- ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling
- Neural Continuous-Discrete State Space Models for Irregularly-Sampled Time Series
- Parallel Square-Root Statistical Linear Regression for Inference in Nonlinear State Space Models
- Robust Automatic Differentiation of Square-Root Kalman Filters via Gramian Differentials
- Kalman Bayesian Transformer
- Deep Continuous-Time State-Space Models for Marked Event Sequences
- Stochastic Integration Based Estimator: Robust Design and Stone Soup Implementation