TL;DR
The MoE-Transformer is a novel dual-domain framework that significantly boosts long-term time-series forecasting by processing data in both time and frequency domains simultaneously. By using Reinforcement Learning (RL) to dynamically route data to specialized "experts" and introducing an "Extended DFT" to fix spectral misalignment, the model achieves over 50% reduction in error while being 60% faster than current state-of-the-art methods.
What Problem Does This Paper Address?
Predicting the future of complex systems—like electrical grids or traffic flow—is notoriously difficult. These systems exhibit "multi-scale" patterns: some behaviors repeat every hour, others every week, and some are non-stationary trends that don't repeat at all.
Current Transformer-based models often struggle with two things:
- Domain Limitation: Most models operate strictly in the time domain, missing the clear periodic signals visible in the frequency domain. Those that do use frequency often suffer from "spectral misalignment," where the frequency grid of the input doesn't match the frequency grid of the full prediction window.
- Static Routing: Modern "Mixture-of-Experts" (MoE) architectures use static gating (like picking the top-k experts). These gates don't adapt well to evolving patterns or different forecasting horizons, often leading to "expert collapse" where only a few sub-networks do all the work.
Methods
The authors introduce the MoE-Transformer, which utilizes three primary innovations:
- Extended Discrete Fourier Transform (Extended DFT): Standard DFTs are calculated based on the input length ($L$). However, the model needs to predict a future window ($T$). The authors calculate an "Extended" spectrum that aligns the input data with the frequency grid of the entire $L+T$ window. This ensures the "notes" the model hears in the input are on the same scale as the "notes" it needs to predict.
- Dual-Domain MoE Architecture: The model splits into two parallel paths: a T-MoE (Time-domain) and an F-MoE (Frequency-domain). Instead of standard feed-forward layers, it uses sets of specialized experts. The frequency path uses complex-valued neural networks to handle the magnitude and phase of the signal.
- RL-Based Dynamic Routing: This is the "brain" of the system. Instead of a simple mathematical gate, the model uses Reinforcement Learning agents (specifically using the ISAC algorithm) to decide which expert should handle which piece of data. The RL reward function doesn't just look at accuracy; it also rewards routing consistency (don't switch experts too erratically) and expert diversity (make sure all experts are learning something useful).
Datasets
The model was evaluated on five gold-standard benchmarks for long-term forecasting:
- ETTh1 & ETTm1: Electricity Transformer Temperature (hourly and 15-min intervals).
- Electricity: Hourly consumption data for 321 clients.
- Traffic: Road occupancy rates from 862 sensors in San Francisco.
- Weather: 21 meteorological indicators collected every 10 minutes.
Key Insights and Ideas
- The Power of Alignment: The Extended DFT alone provided a 5.8% performance gain. It proves that in frequency-domain modeling, how you define your "grid" is just as important as the model architecture itself.
- RL as a Stabilizer: One of the most interesting findings is that RL-based routing outperformed static gating by up to 47.2%. By treating routing as a sequential decision-making process (a Markov Decision Process), the model learns to pick experts that are not just good for the current step, but contribute to a stable, long-term forecasting strategy.
- Specialization Emerges Naturally: In the frequency domain, the RL agent learned to send dominant, low-frequency signals to specific experts while delegating high-frequency "noise" to others. In the time domain, experts specialized in handling trends versus sudden, irregular fluctuations.
Key Results
The MoE-Transformer sets a new bar for the field:
- Accuracy: It achieved MSE reductions of 50.9% to 56.9% compared to top-tier baselines like PatchTST and TimesNet.
- Efficiency: Because it uses "sparse" activation (only turning on the necessary experts), it uses 40% less memory and boasts a 60% reduction in inference latency.
- Scalability: Performance continued to improve as the number of experts increased (tested up to 40), showing that the RL router effectively manages larger "brain" capacities without getting confused.
Limitations and Future Work
While the results are impressive, the paper notes a few areas for growth:
- Univariate Focus: The current framework processes different variables (channels) independently. While this is a common and effective strategy, it doesn't explicitly model the cross-dependencies between, say, temperature and electricity usage.
- Deterministic Output: The model provides a "point forecast." Future iterations could incorporate probabilistic methods to provide a range of uncertainty, which is critical for risk management in power grids.
- Complexity: Implementing dual RL agents alongside a Transformer is computationally heavier during the training phase, even if inference is faster.
Reference: Li, Q., et al. "Learning to Route in Time and Frequency Domains: A Dual-Domain MoE Transformer for Multi-Horizon Forecasting." Research Square Preprint, 2026.
Summary updated