TL;DR
IMRouter addresses "Temporal Pattern Imbalance" (TPI) in LLM-based time series forecasting—a phenomenon where frequent temporal patterns dominate model training, leading to poor performance on rare but critical events. By employing a specialized Mixture-of-Experts (MoE) architecture guided by gradient conflict and density-aware weighting, IMRouter ensures that rare temporal dynamics are adequately represented and aligned with the LLM's semantic space.
What Problem Does This Paper Address?
Large Language Models (LLMs) have recently shown great promise in time series forecasting (TSF) by bridging the gap between numerical data and linguistic prototypes. However, the authors identify a significant hurdle: Temporal Pattern Imbalance (TPI).
In real-world data, certain dynamics (like steady trends) occur much more frequently than others (like sudden oscillations). Because standard models are trained to minimize Mean Squared Error (MSE) across the entire dataset, the optimization process is naturally biased toward these "head" patterns. As shown in the authors' empirical analysis, this leads to a "rich get richer" scenario where the model excels at frequent patterns but fails miserably on the "tail" patterns that occur less often.
Methods
The authors propose IMRouter, a framework designed to balance learning across diverse temporal regimes. It consists of two primary components:
- Pattern-Adaptive Temporal Representation Learning (PATR): Instead of a single shared encoder, PATR uses a Mixture-of-Experts (MoE). To prevent the router from simply sending everything to the "frequent pattern" expert, they introduce two novel regularizers:
- Gradient Conflict Penalty: If two data patches assigned to the same expert produce opposing gradients, it suggests the expert is being forced to learn incompatible dynamics. PATR penalizes these joint assignments.
- Density-Aware Weighting: This increases the relative importance of conflicts involving rare patterns, ensuring the "tail" optimization signals aren't drowned out by the "head."
- Routing-Guided Semantic Alignment (RGSA): LLM-based TSF requires aligning temporal features with text prototypes (e.g., mapping a "spike" to a specific embedding). RGSA uses mutual information maximization to distill the pattern-specific knowledge from the MoE router into the alignment module. This ensures that the distinctiveness of rare patterns is preserved when the data is "translated" for the LLM.
Datasets
The model was evaluated on several widely-used TSF benchmarks:
- Long-term Forecasting: ETT (h1, h2, m1, m2), Weather, Electricity (ECL), and Traffic.
- Short-term Forecasting: The M4 benchmark (100,000 univariate series).
- Few-shot Learning: Evaluated on ETT datasets using only 10% of training data.
The authors used TS2Vec and HDBSCAN clustering on these datasets to quantify TPI, finding that in the Traffic dataset, for example, the top 20% of clusters cover nearly 95% of the samples.
Key Insights and Ideas
- The Gradient as a Signal for Diversity: Rather than relying on manual labels for "rare" vs "frequent," IMRouter looks at gradient conflict. If the model struggles to satisfy two samples simultaneously, they likely belong to different temporal patterns.
- TPI is Ubiquitous: The paper provides a statistical breakdown showing that TPI exists across almost all major TSF benchmarks, measured by Gini coefficients and coverage statistics.
- Alignment is a Bottleneck for Rare Patterns: The authors found that even if an encoder learns a rare pattern, the "cross-modal alignment" (mapping numbers to text) often washes that information away in favor of frequent prototypes. The RGSA module is a direct fix for this "semantic dilution."
Key Results
IMRouter consistently outperformed state-of-the-art (SOTA) baselines like FSCA and CALF:
- Overall Performance: Achieved a 1.4% reduction in MSE and 2.0% reduction in MAE compared to the strongest baseline (CALF) across seven long-term benchmarks.
- Tail Performance: On the most infrequent patterns (Cluster 6 in the Traffic dataset), IMRouter achieved an MSE of ~0.78, while competing methods hovered between 0.85 and 0.93.
- Few-Shot Efficiency: In 10% data scenarios, IMRouter maintained high accuracy, suggesting that its specialized experts learn more efficiently from limited samples of rare patterns.
Limitations and Future Work
- Computational Overhead: The use of MoE and complex gradient-based regularizers during training increases the computational cost compared to simpler linear or single-transformer models.
- Backbone Dependency: The performance is still somewhat tied to the underlying LLM (GPT-2 was used in experiments). Exploring how larger models (like LLaMA 3) interact with TPI would be a natural next step.
- Dynamic TPI: The current work assumes a static distribution of patterns. In non-stationary series where patterns evolve over time, the router might need more dynamic adaptation.
Summary updated