SEMixer: Semantics Enhanced MLP-Mixer for Multiscale Mixing and Long-term Time Series Forecasting

Xu Zhang, Qitong Wang, Peng Wang, Wei Wang

TL;DR

SEMixer is a novel, lightweight deep learning model for long-term time series forecasting that excels at capturing complex, multi-scale patterns. It achieves this by introducing a "Random Attention Mechanism" (RAM) to better understand semantic relationships between time segments and a "Multiscale Progressive Mixing Chain" (MPMC) to efficiently integrate information across different temporal scales, outperforming existing methods in accuracy and efficiency.

What Problem Does This Paper Address?

Accurately forecasting long-term time series (TS) is crucial for many applications, from financial markets to infrastructure monitoring. Time series data often exhibit patterns at various scales – think daily fluctuations versus monthly trends. Effectively modeling these "multiscale temporal dependencies" is key. However, existing methods struggle with:

  • Computational Overhead: Integrating information from multiple scales can be computationally expensive, leading to memory issues and slow training, especially with long historical data.
  • Semantic Gaps: Non-adjacent scales (e.g., hourly vs. yearly) can have very different underlying dynamics, creating "semantic gaps" that are hard to bridge.
  • Noise and Redundancy: Real-world time series are often noisy and contain redundant information, which can dilute important signals and hinder accurate forecasting.

These challenges limit the ability of current models to leverage long historical sequences, which are vital for robust long-term predictions.

Methods

SEMixer builds upon the MLP-Mixer architecture, known for its efficiency in vision tasks, and adapts it for time series forecasting. The core innovations are:

  • Random Attention Mechanism (RAM): Instead of traditional self-attention, which can be computationally intensive and prone to overfitting noise, RAM uses a randomized approach. During training, it randomly samples interaction patterns between time patches. At inference, it aggregates these learned patterns using a dropout ensemble. This allows it to capture diverse semantic relationships between time segments more efficiently and robustly.
  • Multiscale Progressive Mixing Chain (MPMC): This component addresses the semantic gaps and computational burden of multiscale modeling. It progressively mixes information from adjacent scales, starting from the finest scale and moving to coarser ones. This pairwise approach is more memory-efficient than processing all scales simultaneously and helps in better aligning and integrating information across different temporal resolutions.

These two components work together within a "Temporal Mixing Block," which is then stacked to form the MPMC.

Datasets

The authors evaluated SEMixer on a diverse set of 10 public datasets covering various domains:

  • Industry: ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2)
  • Climate: Weather
  • Energy: Electricity, Solar Energy
  • Health: ILI (Influenza-Like Illness)
  • Economy: Exchange rate
  • Traffic: Traffic

Additionally, SEMixer was tested on a large-scale, real-world dataset from the 2025 CCF AlOps Challenge, consisting of 21GB of wireless network data, where it achieved third place.

Key Insights and Ideas

  • Randomness as a Feature: The paper demonstrates that a carefully designed randomized attention mechanism (RAM) can be more effective and efficient than traditional self-attention for time series, especially in handling noise and semantic gaps.
  • Progressive Integration is Key: The MPMC's strategy of progressively mixing scales, rather than all at once, is crucial for managing computational resources and effectively bridging semantic differences between scales.
  • MLP-Mixer's Potential: By adapting MLP-Mixer with RAM and MPMC, the authors show that efficient, non-attention-based architectures can achieve state-of-the-art performance in complex time series tasks.
  • Longer Inputs = Better Forecasts: SEMixer consistently shows improved performance with longer input sequences, highlighting its ability to leverage historical data more effectively than many existing methods.

Key Results

  • Superior Accuracy: SEMixer consistently outperforms 12 advanced baselines (including Transformer, CNN, and linear models) on the 10 public datasets, achieving lower MSE and MAE. For example, it shows a 5-15% lower MSE on average.
  • Efficiency Gains: SEMixer is significantly more memory-efficient and faster to train than many multiscale models, even outperforming single-scale models in some efficiency metrics. For instance, it avoids the "out-of-memory" issues encountered by models like Pathformer on large datasets.
  • Strong Performance on Real-World Data: SEMixer secured third place in the 2025 CCF AlOps Challenge, demonstrating its effectiveness on a large-scale, real-world industrial dataset.
  • Robustness: Ablation studies show that SEMixer is robust to noise, with RAM and MPMC playing key roles in maintaining performance under noisy conditions.

Limitations and Future Work

  • Noise Sensitivity of RAM: While RAM is generally robust, the paper notes that it might introduce noise if the underlying correlations between patches are very weak. A residual connection is incorporated to mitigate this.
  • Hyperparameter Sensitivity: While the paper presents hyperparameter analysis for RAM's disconnection probability, further exploration of optimal settings for MPMC and other components across diverse datasets could be beneficial.
  • Future Work: The authors suggest that their approach could be extended to other time series tasks beyond forecasting and that further research into optimizing the interaction between RAM and MPMC could yield even greater improvements. The code is publicly available, encouraging further research and development.

Summary updated

Abstract

Modeling multiscale patterns is crucial for long-term time series forecasting (TSF). However, redundancy and noise in time series, together with semantic gaps between non-adjacent scales, make the efficient alignment and integration of multi-scale temporal dependencies challenging. To address this, we propose SEMixer, a lightweight multiscale model designed for long-term TSF. SEMixer features two key components: a Random Attention Mechanism (RAM) and a Multiscale Progressive Mixing Chain (MPMC). RAM captures diverse time-patch interactions during training and aggregates them via dropout ensemble at inference, enhancing patch-level semantics and enabling MLP-Mixer to better model multi-scale dependencies. MPMC further stacks RAM and MLP-Mixer in a memory-efficient manner, achieving more effective temporal mixing. It addresses semantic gaps across scales and facilitates better multiscale modeling and forecasting performance. We not only validate the effectiveness of SEMixer on 10 public datasets, but also on the \textit{2025 CCF AlOps Challenge} based on 21GB real wireless network data, where SEMixer achieves third place. The code is available at the link https://github.com/Meteor-Stars/SEMixer.
BibTeX
@inproceedings{zhang2026semixer,
  author = {Xu Zhang and Qitong Wang and Peng Wang and Wei Wang},
  title = {{SEMixer: Semantics Enhanced MLP-Mixer for Multiscale Mixing and Long-term Time Series Forecasting}},
  booktitle = {The Web Conference},
  year = {2026},
  doi = {10.1145/3774904.3792639},
}
Figure