TL;DR
Moirai-MoE moves away from human-defined heuristics in time series foundation models, replacing frequency-based specialization with a data-driven Sparse Mixture of Experts (MoE). By enabling automatic, token-level specialization, the model achieves up to a 17% performance boost over its predecessor and outperforms state-of-the-art models like TimesFM and Chronos while using significantly fewer activated parameters.
What Problem Does This Paper Address?
Building a "universal" time series foundation model is difficult because time series data is incredibly heterogeneous. A model needs to handle everything from hourly electricity usage to yearly economic indicators.
Existing models like the original MOIRAI and TimesFM try to solve this using frequency-level specialization. They use different mathematical "layers" or embeddings depending on whether the data is hourly, daily, or monthly. The authors identify two fatal flaws in this approach:
- Frequency ≠ Pattern: Two different frequencies (e.g., daily and hourly) might show identical patterns, while two series with the same frequency might look completely different.
- Non-stationarity: Real-world data changes over time. A single time series might shift its distribution mid-stream, making a coarse, frequency-based approach too rigid to capture fine-grained changes.
Methods
Moirai-MoE shifts the burden of specialization from the human designer to the model itself.
- Sparse MoE Architecture: Instead of multiple frequency-specific input layers, the model uses a single projection layer. Inside the Transformer, standard Feed-Forward Networks (FFNs) are replaced with MoE layers. Each layer contains 32 "experts," but only the top 2 are activated for any given token, keeping the computational cost low.
- Token-Level Specialization: Because the choice of expert happens at the token level (individual patches of the time series), the model can dynamically switch experts as the patterns in the data evolve.
- Token Clusters as Gating: Standard MoE models use a randomly initialized linear layer to route tokens to experts. Moirai-MoE introduces a novel gating function that uses cluster centroids derived from a pretrained MOIRAI model. This "guides" the experts by grouping similar patterns together based on existing knowledge.
- Decoder-Only Training: The model adopts a generative, autoregressive objective, which is more efficient for training across various context lengths simultaneously.
Datasets
The model was pretrained on the LOTSA archive, a massive collection containing over 27 billion observations across nine domains.
For evaluation, the authors used:
- In-distribution: 29 datasets from the Monash benchmark (e.g., M4, Tourism, Traffic, Weather).
- Zero-shot (Out-of-distribution): 10 diverse datasets not included in the pretraining set, ranging from Turkish Power demand to Jena Weather and BizITObs.
Key Insights and Ideas
- Frequency Invariance: Analysis shows that while shallow layers of the model distinguish between frequencies, the deeper layers converge on similar expert selections. This suggests the model successfully learns "frequency-invariant" representations—abstracting the underlying signal regardless of how often it was sampled.
- Progressive Denoising: The authors observed that shallow layers have very diverse routing preferences (handling the "noise" and local variability), while deep layers focus on a few specific experts. This indicates the model performs a progressive denoising process as the signal moves through the layers.
- Periodicity Awareness: The MoE routing isn't random; it actually tracks the physical properties of the data. In traffic data, the model automatically assigns different experts to handle the "rising," "peak," and "falling" phases of a daily cycle.
Key Results
- Superior Accuracy: MOIRAI-MOE outperformed the original MOIRAI by up to 17% in aggregate MAE.
- Efficiency Giant: It achieved better zero-shot performance than Chronos-L while using 65x fewer activated parameters per token.
- State-of-the-Art: In zero-shot scenarios, MOIRAI-MOE_B (the Base version) achieved the best average performance across 10 datasets, even beating models like TimesFM that had seen some of that data during their own pretraining.
- Inference Speed: Despite the complexity of MoE, the model maintains inference speeds comparable to dense models and is significantly faster than point-wise autoregressive models like Chronos because it processes data in "patches."
Limitations and Future Work
- Expert Underutilization: The authors noted that in the deepest layers, only a few experts (3 out of 32) are consistently chosen. This suggests that the model could be further compressed by pruning these "lazy" experts.
- Computational Requirements: While inference is efficient, training a model with nearly a billion total parameters (even if only 86M are active at once) still requires significant GPU resources.
- Future Directions: The study opens the door for exploring even more sophisticated gating functions and applying MoE to other time-series-adjacent tasks like classification or anomaly detection.
Summary updated