TL;DR
The researchers introduce ci-proto, a new family of time series foundation models that replaces the standard Transformer architecture with Mamba-3 linear recurrences. By alternating linear temporal processing with cross-variate attention, ci-proto achieves state-of-the-art performance on the fev-bench leaderboard while maintaining linear scaling and defining the entire Pareto frontier for accuracy versus both parameter count and inference speed.
What Problem Does This Paper Address?
Time series foundation models (TSFMs) have recently shifted the field toward zero-shot forecasting, but they face a fundamental architectural trade-off. Most current TSFMs rely on Transformers, which allow for parallel training but suffer from quadratic computational costs as sequence lengths grow. Conversely, traditional Recurrent Neural Networks (RNNs) offer linear inference costs but create a sequential bottleneck during training, making them difficult to scale on modern hardware.
The authors tackle this "efficiency vs. expressivity" gap, specifically for multivariate forecasting. They aim to build a model that can handle long histories and complex interactions between variables without the massive memory and compute overhead of self-attention.
Methods
The core innovation of ci-proto is its use of Mamba-3 as a temporal backbone. Mamba-3 utilizes linear state recurrences, which allow for:
- Chunk-wise parallel-in-time computation: This enables efficient training similar to Transformers.
- Linear scaling: Inference costs grow linearly with sequence length, unlike the quadratic growth of Transformers.
- Enhanced Dynamics: Mamba-3 improves upon previous versions by using input-dependent $2 \times 2$ rotation matrices, which better track state over time.
The architecture alternates between two types of layers:
- Time Mixer: A Mamba-3 block that processes each variable's temporal sequence independently.
- Variate Mixer: An asymmetric group attention mechanism that allows variables to "talk" to each other. Crucially, it supports different roles: targets can attend to covariates, but covariates (which are often known or fixed) don't necessarily need to attend to targets.
The model is trained to predict 9 equally spaced quantiles (0.1 to 0.9) using a pinball loss, enabling robust probabilistic forecasting.
Datasets
- Pre-training: A massive corpus including subsets of the Chronos training data, GIFT-Eval, and a significant amount of synthetic data generated via Gaussian processes and structural causal models to simulate multivariate relationships.
- Evaluation: The models were benchmarked on fev-bench, a comprehensive suite of 100 forecasting tasks covering diverse domains and settings (past-only vs. future-known covariates).
Key Insights and Ideas
- Linear Recurrences are SOTA-Ready: The paper demonstrates that you don't need quadratic self-attention to achieve state-of-the-art results in time series. ci-proto XL outperforms heavyweights like TimesFM and Chronos.
- The Pareto Frontier Dominance: Perhaps the most striking finding is that the five ci-proto variants (XS to XL) occupy the entire Pareto frontier. Whether you care about the lowest error, the fewest parameters, or the fastest runtime, a ci-proto model is likely the optimal choice.
- Extreme Long-Range Recall: Because of its linear scaling, the authors successfully fine-tuned ci-proto XL on contexts of up to 1,048,576 steps. In a minute-resolution electricity consumption task, the model was able to recall a specific "power outage" event from nearly 4 years prior to improve its forecast—a feat virtually impossible for Transformer-based models due to memory constraints.
Key Results
- Performance:
ci-proto XLset a new state of the art onfev-benchwith a Mean Absolute Scaled Error (MASE) of 1.387 and a Scaled Quantile Loss (SQL) of 1.160. - Efficiency: Even the smallest variant,
ci-proto XS(3.5M parameters), achieves better accuracy than much larger Transformer models while running significantly faster. - Inference Speed: As shown in Figure 3, ci-proto models provide a superior accuracy-to-latency trade-off compared to existing non-linear RNNs and Transformers.
Limitations and Future Work
- Test-Time Adaptation (TTA): The authors utilize a suite of TTA techniques (like trend-differencing and sign-flip augmentation). While these are standard in the latest TSFMs, it would be interesting to see how much of the "foundation" knowledge is in the weights versus the post-processing logic.
- Variate Scaling: While temporal processing is linear, the cross-variate attention still scales with the number of variables. For datasets with thousands of variables, this could become a new bottleneck.
- Future Directions: The authors suggest further scaling and exploring even more complex multivariate interactions as promising next steps.
Summary updated