TL;DR
This paper introduces MoE-Encoder, a novel module that enhances pre-trained time series models for multivariate forecasting. By using covariates to guide a sparse Mixture-of-Experts (MoE) layer, it enables efficient, privacy-preserving adaptation in federated settings, significantly improving forecasting accuracy.
What Problem Does This Paper Address?
Accurate forecasting of multivariate time series is crucial for systems like power grids, where decisions depend on predicting energy consumption, supply-demand imbalances, and system loads. These time series are inherently complex, with interdependencies between variables and often siloed data across different regions due to privacy and regulatory constraints. Traditional forecasting methods struggle with this complexity and generalization. While recent advancements in pre-trained models offer promise, their zero-shot performance on specific tasks is limited, and they often require full model retraining or lack privacy-preserving capabilities for federated learning. This paper tackles the challenge of achieving accurate, generalized, and privacy-aware multivariate forecasting in decentralized environments.
Methods
The core innovation is the MoE-Encoder module, which is inserted between the tokenization and encoding layers of existing time series forecasting models (specifically, they use Chronos as a backbone). This module comprises:
- Covariate-Guided Expert Selection: Auxiliary covariates (like time of day, region ID, or weather) are used to dynamically route input data to specific "expert" sub-networks within the MoE layer. This transforms the multivariate forecasting problem into a more manageable, expert-guided univariate task for each expert.
- Sparse Mixture-of-Experts (MoE): A sparse MoE layer is employed, meaning only a subset of experts are activated for any given input token. This conditional computation makes the model more efficient.
- Federated Compatibility: The MoE-Encoder is designed for federated learning. Clients can train only their MoE-Encoder locally, keeping the main backbone model frozen. This allows for personalized adaptation without sharing raw data. Only the MoE-Encoder parameters (gating function and selected experts) need to be transmitted for sharing, significantly reducing communication overhead.
- Parameter-Efficient Fine-tuning: For federated adaptation, they leverage LoRA (Low-Rank Adaptation) on the MoE-Encoder, further minimizing the number of parameters that need to be updated or shared.
Datasets
The authors evaluate their approach on five publicly available multivariate electricity market price datasets from the Zenodo repository:
- electricity-BE
- electricity-DE
- electricity-FR
- electricity-NP
- electricity-PJM
These datasets are chosen for their diverse consumption patterns, seasonal structures, and covariate contexts, making them suitable for testing cross-domain generalization.
Key Insights and Ideas
- Covariates as Intelligent Routers: The most significant insight is how covariates can be used not just as input features but as intelligent "routers" to guide expert selection. This allows the model to adapt its internal structure based on context, leading to more specialized and accurate predictions.
- Federated Learning with Modular Experts: The MoE-Encoder provides a powerful framework for federated learning by enabling modularity. Clients can specialize their forecasting capabilities without compromising data privacy, and only lightweight parameter updates are needed for adaptation.
- Bridging the Gap Between Centralized and Decentralized: The approach effectively bridges the gap between powerful centralized models and the practical constraints of decentralized, privacy-sensitive environments. It offers a way to leverage the benefits of large pre-trained models in scenarios where data cannot be pooled.
- Expert Redundancy for Robustness: The experiments on covariate perturbations suggest that having multiple experts, even if not all are activated, can improve the model's robustness to missing or noisy covariate information.
Key Results
- Improved Accuracy: MoE-Encoder consistently outperforms strong baselines (Chronos, PatchTST, CITRAS, COSMIC) across all five electricity datasets in terms of Weighted Quantile Loss (WQL) and Mean Absolute Scaled Error (MASE). It achieved the lowest WQL on three datasets and competitive MASE scores.
- Federated Adaptation Efficiency: In simulated federated environments, transferring only MoE-Encoder parameters allowed for efficient adaptation to new regions with minimal performance degradation.
- Reduced Communication Overhead: MoE-Encoder reduces communication overhead by 42.3% compared to full fine-tuning in a federated setting.
- Gating Strategy Matters: Covariate-fixed gating (the proposed method) significantly outperforms softmax top-k gating and random expert assignment, highlighting the importance of informed expert selection.
- Optimal Expert Number: Ablation studies suggest an intermediate number of experts (around 8) strikes a balance between diversity and robustness, with performance plateauing or slightly degrading with too many experts.
Limitations and Future Work
- Expert Number Optimization: While an intermediate number of experts is found to be optimal, the paper notes that further research could explore more dynamic or adaptive ways to determine the number of experts.
- Generalization to Other Tasks: The authors suggest future work will explore extending MoE-Encoder to other time series tasks beyond price forecasting.
- Advanced Privacy Mechanisms: Investigating more sophisticated privacy-preserving mechanisms for expert sharing in federated environments is also a future direction.
- Real-world Federated Deployment: While simulated, real-world deployment in diverse federated settings with varying data distributions and client capabilities would be a valuable next step.
Summary updated