TL;DR
xLSTM-Mixer introduces a novel architecture for multivariate time series forecasting by combining the expressive power of Extended Long Short-Term Memory (xLSTM) with multi-stage mixing architectures. By refining a simple linear forecast through an sLSTM stack that strides across variates, it achieves state-of-the-art long-term forecasting accuracy. Crucially, it delivers these results with a tiny memory footprint, requiring one to two orders of magnitude less GPU memory than dominant Transformer-based models.
What Problem Does This Paper Address?
Predicting the future of multivariate time series data requires capturing two distinct types of patterns: how a single variable changes over time (temporal dependencies) and how different variables interact with one another (cross-variate dependencies).
Currently, large Transformer models dominate this field due to their ability to handle long-range dependencies. However, Transformers suffer from a fundamental flaw: their attention mechanism scales quadratically with sequence length and the number of variables, making them incredibly memory-intensive and difficult to deploy on edge devices or embedded systems.
Traditional Recurrent Neural Networks (RNNs) and LSTMs scale linearly and are highly efficient, but historically struggled to retain information over long horizons. While the recently proposed xLSTM architecture fixes these recurrent limitations using exponential gating and memory mixing, it had not yet been successfully adapted to capture both temporal and cross-variate relationships in complex multivariate time series data.
Methods
The authors introduce xLSTM-Mixer, which breaks the forecasting problem down into three distinct mixing stages:
- Time Mixing (Initial Linear Forecast): The raw time series is first normalized using Reversible Instance Normalization (RevIN). Then, a simple
NLinearmodel processes each variable (channel) independently. This shared linear layer acts as a regularizer and provides a baseline temporal forecast. - Joint Mixing (sLSTM Refinement): The initial forecast is up-projected into a higher hidden dimension. Instead of feeding time steps sequentially into a recurrent network, xLSTM-Mixer transposes the data and strides its sLSTM (scalar-memory LSTM) stack across the variate axis. A learnable "soft prompt" token ($\eta$) is prepended to the sequence to initialize the sLSTM's memory state based on specific dataset characteristics.
- View Mixing (Multi-View Reconciliation): To prevent the model from overfitting to a specific ordering of variables, the architecture processes two views simultaneously: the original variate sequence and a reversed variate sequence. The latent forecasts from both views are finally reconciled via a learned fully connected layer (
FCview) to produce the final prediction.
Datasets
The model was rigorously evaluated across three distinct tasks using standard benchmarks:
- Long-Term Forecasting: Weather, Electricity, Traffic, and the Electricity Transformer Temperature (ETT) datasets (ETTh1, ETTh2, ETTm1, ETTm2).
- Probabilistic Forecasting: Evaluated on the comprehensive GIFT-Eval benchmark, which spans diverse domains, frequencies, and horizons.
- Time Series Classification: Standard classification benchmarks including EthanolConcentration, FaceDetection, Handwriting, and JapaneseVowels.
Key Insights and Ideas
- Striding Over Variates, Not Time: Standard recurrent approaches process time steps sequentially. xLSTM-Mixer flips this paradigm by treating entire variable histories as tokens and recurrently stepping through the channels. This allows the sLSTM blocks to directly mix cross-channel information.
- The Crucial Role of sLSTM's Scalar Memory: The authors demonstrate that for long-horizon forecasting, tracking latent phases (e.g., cyclical ups and downs) is critical. The sLSTM block retains hidden-to-hidden recurrences that allow it to dynamically overwrite or retain memory. Parallelizable alternatives like mLSTM or Mamba lack this specific "memory mixing" capability and perform significantly worse on these tasks.
- Implicit Ensembling via Reversal: Reversing the channel order during the Multi-View Mixing stage acts as a powerful multi-task regularizer. It effectively ensembles different variate orderings while sharing weights, leading to more robust representations.
Key Results
- State-of-the-Art Forecasting: xLSTM-Mixer consistently outperformed competitive baselines (including PatchTST, iTransformer, and TimeMixer), achieving the best performance in 11 out of 28 cases for MSE and 16 out of 28 cases for MAE across long-term benchmarks.
- GIFT-Eval Success: On the massive
GIFT-Evalbenchmark, xLSTM-Mixer ranked 2nd out of 39 models overall, making it the top-performing purely supervised model (outperforming many heavily pre-trained foundation models). - Extreme Efficiency: As shown in Figure 1, xLSTM-Mixer scales exceptionally well. While Transformer-based models experience explosive GPU memory growth as lookback windows increase, xLSTM-Mixer maintains a near-flat, low-memory footprint while remaining highly computational-speed competitive.
Limitations and Future Work
Despite its strong performance, xLSTM-Mixer has a few limitations:
- Uniform Grid Assumption: The model assumes all variables are sampled on a uniform time grid and cannot natively handle missing or irregularly spaced timestamps without prior imputation.
- High-Dimensional Scaling: Because the sequential axis is tied to the number of channels, datasets with an extremely large number of variables (thousands+) could cause a computational bottleneck.
Future Directions: The authors suggest exploring adaptive variate grouping to handle massive channel counts, integrating continuous-time embeddings for irregular data, and extending the framework to downstream tasks like anomaly detection and data imputation.
Summary updated