TL;DR
SEMixer is a novel, lightweight deep learning model for long-term time series forecasting that excels at capturing complex, multi-scale patterns. It achieves this by introducing a "Random Attention Mechanism" (RAM) to better understand semantic relationships between time segments and a "Multiscale Progressive Mixing Chain" (MPMC) to efficiently integrate information across different temporal scales, outperforming existing methods in accuracy and efficiency.
What Problem Does This Paper Address?
Accurately forecasting long-term time series (TS) is crucial for many applications, from financial markets to infrastructure monitoring. Time series data often exhibit patterns at various scales – think daily fluctuations versus monthly trends. Effectively modeling these "multiscale temporal dependencies" is key. However, existing methods struggle with:
- Computational Overhead: Integrating information from multiple scales can be computationally expensive, leading to memory issues and slow training, especially with long historical data.
- Semantic Gaps: Non-adjacent scales (e.g., hourly vs. yearly) can have very different underlying dynamics, creating "semantic gaps" that are hard to bridge.
- Noise and Redundancy: Real-world time series are often noisy and contain redundant information, which can dilute important signals and hinder accurate forecasting.
These challenges limit the ability of current models to leverage long historical sequences, which are vital for robust long-term predictions.
Methods
SEMixer builds upon the MLP-Mixer architecture, known for its efficiency in vision tasks, and adapts it for time series forecasting. The core innovations are:
- Random Attention Mechanism (RAM): Instead of traditional self-attention, which can be computationally intensive and prone to overfitting noise, RAM uses a randomized approach. During training, it randomly samples interaction patterns between time patches. At inference, it aggregates these learned patterns using a dropout ensemble. This allows it to capture diverse semantic relationships between time segments more efficiently and robustly.
- Multiscale Progressive Mixing Chain (MPMC): This component addresses the semantic gaps and computational burden of multiscale modeling. It progressively mixes information from adjacent scales, starting from the finest scale and moving to coarser ones. This pairwise approach is more memory-efficient than processing all scales simultaneously and helps in better aligning and integrating information across different temporal resolutions.
These two components work together within a "Temporal Mixing Block," which is then stacked to form the MPMC.
Datasets
The authors evaluated SEMixer on a diverse set of 10 public datasets covering various domains:
- Industry: ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2)
- Climate: Weather
- Energy: Electricity, Solar Energy
- Health: ILI (Influenza-Like Illness)
- Economy: Exchange rate
- Traffic: Traffic
Additionally, SEMixer was tested on a large-scale, real-world dataset from the 2025 CCF AlOps Challenge, consisting of 21GB of wireless network data, where it achieved third place.
Key Insights and Ideas
- Randomness as a Feature: The paper demonstrates that a carefully designed randomized attention mechanism (RAM) can be more effective and efficient than traditional self-attention for time series, especially in handling noise and semantic gaps.
- Progressive Integration is Key: The MPMC's strategy of progressively mixing scales, rather than all at once, is crucial for managing computational resources and effectively bridging semantic differences between scales.
- MLP-Mixer's Potential: By adapting MLP-Mixer with RAM and MPMC, the authors show that efficient, non-attention-based architectures can achieve state-of-the-art performance in complex time series tasks.
- Longer Inputs = Better Forecasts: SEMixer consistently shows improved performance with longer input sequences, highlighting its ability to leverage historical data more effectively than many existing methods.
Key Results
- Superior Accuracy: SEMixer consistently outperforms 12 advanced baselines (including Transformer, CNN, and linear models) on the 10 public datasets, achieving lower MSE and MAE. For example, it shows a 5-15% lower MSE on average.
- Efficiency Gains: SEMixer is significantly more memory-efficient and faster to train than many multiscale models, even outperforming single-scale models in some efficiency metrics. For instance, it avoids the "out-of-memory" issues encountered by models like Pathformer on large datasets.
- Strong Performance on Real-World Data: SEMixer secured third place in the 2025 CCF AlOps Challenge, demonstrating its effectiveness on a large-scale, real-world industrial dataset.
- Robustness: Ablation studies show that SEMixer is robust to noise, with RAM and MPMC playing key roles in maintaining performance under noisy conditions.
Limitations and Future Work
- Noise Sensitivity of RAM: While RAM is generally robust, the paper notes that it might introduce noise if the underlying correlations between patches are very weak. A residual connection is incorporated to mitigate this.
- Hyperparameter Sensitivity: While the paper presents hyperparameter analysis for RAM's disconnection probability, further exploration of optimal settings for MPMC and other components across diverse datasets could be beneficial.
- Future Work: The authors suggest that their approach could be extended to other time series tasks beyond forecasting and that further research into optimizing the interaction between RAM and MPMC could yield even greater improvements. The code is publicly available, encouraging further research and development.
Summary updated