FSMLP: Modelling Channel Dependencies With Simplex Theory Based Multi-Layer Perceptions In Frequency Domain

Zhengnan Li, Haoxuan Li, Hao Wang, Jun Fang, Duoyin Li, Yunxiao Qin

Abstract

Time series forecasting (TSF) plays a crucial role in various domains, including web data analysis, energy consumption prediction, and weather forecasting. While Multi-Layer Perceptrons (MLPs) are lightweight and effective for capturing temporal dependencies, they are prone to overfitting when used to model inter-channel dependencies. In this paper, we investigate the overfitting problem in channel-wise MLPs using Rademacher complexity theory, revealing that extreme values in time series data exacerbate this issue. To mitigate this issue, we introduce a novel Simplex-MLP layer, where the weights are constrained within a standard simplex. This strategy encourages the model to learn simpler patterns and thereby reducing overfitting to extreme values. Based on the Simplex-MLP layer, we propose a novel Frequency Simplex MLP (FSMLP) framework for time series forecasting, comprising of two kinds of modules: Simplex Channel-Wise MLP (SCWM) and Frequency Temporal MLP (FTM). The SCWM effectively leverages the Simplex-MLP to capture inter-channel dependencies, while the FTM is a simple yet efficient temporal MLP designed to extract temporal information from the data. Our theoretical analysis shows that the upper bound of the Rademacher Complexity for Simplex-MLP is lower than that for standard MLPs. Moreover, we validate our proposed method on seven benchmark datasets, demonstrating significant improvements in forecasting accuracy and efficiency, while also showcasing superior scalability. Additionally, we demonstrate that Simplex-MLP can improve other methods that use channel-wise MLP to achieve less overfitting and improved performance.

Addressing Overfitting in Channel-Mixing

In multivariate time series forecasting, researchers generally choose between two paths: channel-independent methods (which treat each variable separately) and channel-mix methods (which model the correlations between variables). While channel-mixing is essential for capturing complex relationships—such as how temperature fluctuations affect energy load—it is notoriously prone to overfitting.

The authors identify a specific culprit: extreme values. Using Rademacher complexity theory, they demonstrate that standard MLPs have high "capacity," allowing them to memorize noise and outliers rather than learning generalizable patterns. When an MLP tries to mix channels containing these extreme values, the weights can explode, leading to poor performance on unseen data. This is particularly problematic in high-dimensional datasets where the ratio of parameters to data points is high.

The FSMLP Framework

The core innovation of this work is the Simplex-MLP layer. Instead of allowing weights to be any real number, the authors force them to lie within a Standard $n$-simplex. This geometric constraint ensures that all weights are non-negative and sum to exactly one.

The FSMLP architecture and the Simplex-MLP layer compared to standard MLP.

Key Components:

  1. Geometric Constraint: By constraining the weights, the model's Rademacher complexity is mathematically bounded at a much lower level than a standard MLP. This prevents the model from assigning disproportionately large weights to noisy features or outliers.
  2. Logarithmic Transformation: To map unconstrained weights into the simplex, the authors utilize a logarithmic transformation ($log(|W| + 1)$) followed by normalization. This approach dampens the gradients for large weights, ensuring more stable optimization.
  3. Frequency Domain Processing: The FSMLP framework converts time-series data into the frequency domain using the Discrete Cosine Transform (DCT). It then employs two specialized modules:
  4. SCWM (Simplex Channel-Wise MLP): Utilizes the simplex constraint to learn robust inter-channel relationships.
  5. FTM (Frequency Temporal MLP): A standard MLP designed to capture temporal patterns within the frequency components.
  6. Hybrid Loss: The model is trained using a combination of Mean Squared Error (MSE) in the time domain and Mean Absolute Error (MAE) in the frequency domain, which handles varying magnitudes of frequency components more robustly than MSE alone.

Experimental Evaluation

The authors evaluated FSMLP on seven widely recognized public benchmarks, including the ETT suite (Electricity Transformer Temperature), Traffic, ECL (Electricity Consumption Load), and Weather datasets. These represent a mix of granularities and varying numbers of channels (from 7 to 862).

Training curves showing MSE over epochs. FSMLP (Ours) maintains stability while models like Autoformer exhibit significant overfitting.

Key Results:

  • Performance: FSMLP consistently outperformed state-of-the-art (SOTA) models like iTransformer, PatchTST, and FreTS. On the ETTm1 dataset, FSMLP achieved an average MSE of 0.365, compared to iTransformer’s 0.407.
  • Efficiency: FSMLP is remarkably fast. In inference tests, it was often the fastest model, outperforming iTransformer and being orders of magnitude faster than attention-based models like Autoformer.
  • Scalability: The model maintains high accuracy even when trained on only 20% of the available data, suggesting it learns the underlying data distribution much faster than its competitors.
  • Plug-and-Play Potential: Replacing standard MLPs with Simplex-MLPs in other architectures like Autoformer and TSMixer resulted in significant performance boosts, with improvements up to 143% in specific scenarios.

Conclusion and Future Work

FSMLP demonstrates that "simplicity is stability." By providing a rigorous mathematical justification through Rademacher complexity, the authors show why geometric constraints lead to better generalization in the presence of outliers. While FSMLP shows dominant performance on periodic datasets, future work could explore how the model handles purely non-stationary or stochastic data where frequency components might be less informative. Additionally, applying Simplex-MLP to domains beyond time series, such as computer vision or natural language processing, remains a promising avenue for research.

Summary updated

Abstract

Time series forecasting (TSF) plays a crucial role in various domains, including web data analysis, energy consumption prediction, and weather forecasting. While Multi-Layer Perceptrons (MLPs) are lightweight and effective for capturing temporal dependencies, they are prone to overfitting when used to model inter-channel dependencies. In this paper, we investigate the overfitting problem in channel-wise MLPs using Rademacher complexity theory, revealing that extreme values in time series data exacerbate this issue. To mitigate this issue, we introduce a novel Simplex-MLP layer, where the weights are constrained within a standard simplex. This strategy encourages the model to learn simpler patterns and thereby reducing overfitting to extreme values. Based on the Simplex-MLP layer, we propose a novel \textbf{F}requency \textbf{S}implex \textbf{MLP} (FSMLP) framework for time series forecasting, comprising of two kinds of modules: \textbf{S}implex \textbf{C}hannel-\textbf{W}ise MLP (SCWM) and \textbf{F}requency \textbf{T}emporal \textbf{M}LP (FTM). The SCWM effectively leverages the Simplex-MLP to capture inter-channel dependencies, while the FTM is a simple yet efficient temporal MLP designed to extract temporal information from the data. Our theoretical analysis shows that the upper bound of the Rademacher Complexity for Simplex-MLP is lower than that for standard MLPs. Moreover, we validate our proposed method on seven benchmark datasets, demonstrating significant improvements in forecasting accuracy and efficiency, while also showcasing superior scalability. Additionally, we demonstrate that Simplex-MLP can improve other methods that use channel-wise MLP to achieve less overfitting and improved performance. Code are available \href{https://github.com/FMLYD/FSMLP}{\textcolor{red}{here}}.
BibTeX
@misc{li2024fsmlp,
  author = {Zhengnan Li and Haoxuan Li and Hao Wang and Jun Fang and Duoyin Li and Yunxiao Qin},
  title = {{FSMLP: Modelling Channel Dependencies With Simplex Theory Based Multi-Layer Perceptions In Frequency Domain}},
  year = {2024},
  eprint = {2412.01654},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.01654},
}
Figure