Time Flies When You Parallelise: Pareto-efficient Time Series Foundation Models with Linear Recurrences

Torben Berndt, Lingyi Yang, Federico Nardi, Yi Heng Lim, Aj Shears, Benjamin WalkerarXiv 2026 · arXiv

TL;DR

The researchers introduce ci-proto, a new family of time series foundation models that replaces the standard Transformer architecture with Mamba-3 linear recurrences. By alternating linear temporal processing with cross-variate attention, ci-proto achieves state-of-the-art performance on the fev-bench leaderboard while maintaining linear scaling and defining the entire Pareto frontier for accuracy versus both parameter count and inference speed.

What Problem Does This Paper Address?

Time series foundation models (TSFMs) have recently shifted the field toward zero-shot forecasting, but they face a fundamental architectural trade-off. Most current TSFMs rely on Transformers, which allow for parallel training but suffer from quadratic computational costs as sequence lengths grow. Conversely, traditional Recurrent Neural Networks (RNNs) offer linear inference costs but create a sequential bottleneck during training, making them difficult to scale on modern hardware.

The authors tackle this "efficiency vs. expressivity" gap, specifically for multivariate forecasting. They aim to build a model that can handle long histories and complex interactions between variables without the massive memory and compute overhead of self-attention.

Methods

The core innovation of ci-proto is its use of Mamba-3 as a temporal backbone. Mamba-3 utilizes linear state recurrences, which allow for:

  • Chunk-wise parallel-in-time computation: This enables efficient training similar to Transformers.
  • Linear scaling: Inference costs grow linearly with sequence length, unlike the quadratic growth of Transformers.
  • Enhanced Dynamics: Mamba-3 improves upon previous versions by using input-dependent $2 \times 2$ rotation matrices, which better track state over time.

The architecture alternates between two types of layers:

  1. Time Mixer: A Mamba-3 block that processes each variable's temporal sequence independently.
  2. Variate Mixer: An asymmetric group attention mechanism that allows variables to "talk" to each other. Crucially, it supports different roles: targets can attend to covariates, but covariates (which are often known or fixed) don't necessarily need to attend to targets.

The model is trained to predict 9 equally spaced quantiles (0.1 to 0.9) using a pinball loss, enabling robust probabilistic forecasting.

Datasets

  • Pre-training: A massive corpus including subsets of the Chronos training data, GIFT-Eval, and a significant amount of synthetic data generated via Gaussian processes and structural causal models to simulate multivariate relationships.
  • Evaluation: The models were benchmarked on fev-bench, a comprehensive suite of 100 forecasting tasks covering diverse domains and settings (past-only vs. future-known covariates).

Key Insights and Ideas

  • Linear Recurrences are SOTA-Ready: The paper demonstrates that you don't need quadratic self-attention to achieve state-of-the-art results in time series. ci-proto XL outperforms heavyweights like TimesFM and Chronos.
  • The Pareto Frontier Dominance: Perhaps the most striking finding is that the five ci-proto variants (XS to XL) occupy the entire Pareto frontier. Whether you care about the lowest error, the fewest parameters, or the fastest runtime, a ci-proto model is likely the optimal choice.
  • Extreme Long-Range Recall: Because of its linear scaling, the authors successfully fine-tuned ci-proto XL on contexts of up to 1,048,576 steps. In a minute-resolution electricity consumption task, the model was able to recall a specific "power outage" event from nearly 4 years prior to improve its forecast—a feat virtually impossible for Transformer-based models due to memory constraints.
Figure 2: Mean scaled quantile loss (SQL) on fev-bench versus parameter count. Lower SQL is better. The horizontal axis is in logarithmic scale. The dashed line marks the Pareto frontier formed by the five ci-proto variants.

Key Results

  • Performance: ci-proto XL set a new state of the art on fev-bench with a Mean Absolute Scaled Error (MASE) of 1.387 and a Scaled Quantile Loss (SQL) of 1.160.
  • Efficiency: Even the smallest variant, ci-proto XS (3.5M parameters), achieves better accuracy than much larger Transformer models while running significantly faster.
  • Inference Speed: As shown in Figure 3, ci-proto models provide a superior accuracy-to-latency trade-off compared to existing non-linear RNNs and Transformers.
Figure 1: Arithmetic mean errors on fev-bench. ci-proto XL leads in both point (MASE) and probabilistic (SQL) forecasting.

Limitations and Future Work

  • Test-Time Adaptation (TTA): The authors utilize a suite of TTA techniques (like trend-differencing and sign-flip augmentation). While these are standard in the latest TSFMs, it would be interesting to see how much of the "foundation" knowledge is in the weights versus the post-processing logic.
  • Variate Scaling: While temporal processing is linear, the cross-variate attention still scales with the number of variables. For datasets with thousands of variables, this could become a new bottleneck.
  • Future Directions: The authors suggest further scaling and exploring even more complex multivariate interactions as promising next steps.

Summary updated

Abstract

Time series foundation models have demonstrated strong zero-shot forecasting performance, but computational efficiency remains a central architectural challenge. Temporal self-attention incurs quadratic cost in the number of temporal tokens, while nonlinear recurrent state transitions restrict parallel computation along the temporal dimension. We introduce ci-proto, a family of time series foundation models based on Mamba-3, whose linear state recurrences support chunk-wise parallel-in-time computation during training, and inference costs that scale linearly with sequence length. The architecture alternates recurrent temporal processing with cross-variate attention, supporting multivariate probabilistic forecasting with past-only and future-known covariates. Across fev-bench's 100 forecasting tasks, ci-proto XL sets a new state of the art among time series foundation models. The five ci-proto variants also make up the entire Pareto frontier for mean scaled quantile loss and mean absolute scaled error against both parameter count and median runtime.
BibTeX
@misc{berndt2026time,
  author = {Torben Berndt and Lingyi Yang and Federico Nardi and Yi Heng Lim and AJ Shears and Benjamin Walker},
  title = {{Time Flies When You Parallelise: Pareto-efficient Time Series Foundation Models with Linear Recurrences}},
  year = {2026},
  eprint = {2611.16934},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16934},
}
Figure