Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts

Xu Liu, Juncheng Liu, Gerald Woo, Taha Aksu, Yuxuan Liang, Roger Zimmermann, Chenghao Liu, Silvio Savarese, Caiming Xiong, Doyen Sahoo

TL;DR

Moirai-MoE moves away from human-defined heuristics in time series foundation models, replacing frequency-based specialization with a data-driven Sparse Mixture of Experts (MoE). By enabling automatic, token-level specialization, the model achieves up to a 17% performance boost over its predecessor and outperforms state-of-the-art models like TimesFM and Chronos while using significantly fewer activated parameters.

What Problem Does This Paper Address?

Building a "universal" time series foundation model is difficult because time series data is incredibly heterogeneous. A model needs to handle everything from hourly electricity usage to yearly economic indicators.

Existing models like the original MOIRAI and TimesFM try to solve this using frequency-level specialization. They use different mathematical "layers" or embeddings depending on whether the data is hourly, daily, or monthly. The authors identify two fatal flaws in this approach:

  1. Frequency ≠ Pattern: Two different frequencies (e.g., daily and hourly) might show identical patterns, while two series with the same frequency might look completely different.
  2. Non-stationarity: Real-world data changes over time. A single time series might shift its distribution mid-stream, making a coarse, frequency-based approach too rigid to capture fine-grained changes.
Challenges in frequency-based specialization: similar patterns across frequencies and high variance within a single frequency.

Methods

Moirai-MoE shifts the burden of specialization from the human designer to the model itself.

  • Sparse MoE Architecture: Instead of multiple frequency-specific input layers, the model uses a single projection layer. Inside the Transformer, standard Feed-Forward Networks (FFNs) are replaced with MoE layers. Each layer contains 32 "experts," but only the top 2 are activated for any given token, keeping the computational cost low.
  • Token-Level Specialization: Because the choice of expert happens at the token level (individual patches of the time series), the model can dynamically switch experts as the patterns in the data evolve.
  • Token Clusters as Gating: Standard MoE models use a randomly initialized linear layer to route tokens to experts. Moirai-MoE introduces a novel gating function that uses cluster centroids derived from a pretrained MOIRAI model. This "guides" the experts by grouping similar patterns together based on existing knowledge.
  • Decoder-Only Training: The model adopts a generative, autoregressive objective, which is more efficient for training across various context lengths simultaneously.
The Moirai-MoE architecture replaces frequency-specific projections with a single projection and utilizes a Sparse Mixture of Experts (MoE) within the Transformer layers.

Datasets

The model was pretrained on the LOTSA archive, a massive collection containing over 27 billion observations across nine domains.

For evaluation, the authors used:

  • In-distribution: 29 datasets from the Monash benchmark (e.g., M4, Tourism, Traffic, Weather).
  • Zero-shot (Out-of-distribution): 10 diverse datasets not included in the pretraining set, ranging from Turkish Power demand to Jena Weather and BizITObs.

Key Insights and Ideas

  • Frequency Invariance: Analysis shows that while shallow layers of the model distinguish between frequencies, the deeper layers converge on similar expert selections. This suggests the model successfully learns "frequency-invariant" representations—abstracting the underlying signal regardless of how often it was sampled.
  • Progressive Denoising: The authors observed that shallow layers have very diverse routing preferences (handling the "noise" and local variability), while deep layers focus on a few specific experts. This indicates the model performs a progressive denoising process as the signal moves through the layers.
  • Periodicity Awareness: The MoE routing isn't random; it actually tracks the physical properties of the data. In traffic data, the model automatically assigns different experts to handle the "rising," "peak," and "falling" phases of a daily cycle.

Key Results

  • Superior Accuracy: MOIRAI-MOE outperformed the original MOIRAI by up to 17% in aggregate MAE.
  • Efficiency Giant: It achieved better zero-shot performance than Chronos-L while using 65x fewer activated parameters per token.
  • State-of-the-Art: In zero-shot scenarios, MOIRAI-MOE_B (the Base version) achieved the best average performance across 10 datasets, even beating models like TimesFM that had seen some of that data during their own pretraining.
  • Inference Speed: Despite the complexity of MoE, the model maintains inference speeds comparable to dense models and is significantly faster than point-wise autoregressive models like Chronos because it processes data in "patches."

Limitations and Future Work

  • Expert Underutilization: The authors noted that in the deepest layers, only a few experts (3 out of 32) are consistently chosen. This suggests that the model could be further compressed by pruning these "lazy" experts.
  • Computational Requirements: While inference is efficient, training a model with nearly a billion total parameters (even if only 86M are active at once) still requires significant GPU resources.
  • Future Directions: The study opens the door for exploring even more sophisticated gating functions and applying MoE to other time-series-adjacent tasks like classification or anomaly detection.

Summary updated

BibTeX
@inproceedings{liu2025moirai,
  author = {Xu Liu and Juncheng Liu and Gerald Woo and Taha Aksu and Yuxuan Liang and Roger Zimmermann and Chenghao Liu and Silvio Savarese and Caiming Xiong and Doyen Sahoo},
  title = {{Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=SrEOUSyJcR},
}
Figure