Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Equality

Mingtao Zhang, Guoli Yang, Zhanxing Zhu, Mengzhu Wang, Xiaoying BaiarXiv 2025 · DOI · arXiv

Introduction

Attention mechanisms have revolutionized multivariate time series forecasting by capturing complex spatio-temporal dependencies. However, their standard implementation suffers from $O(N^2)$ computational and memory complexity, where $N$ is the sequence length or number of nodes. This quadratic bottleneck hinders scalability for long-range dependencies or large-scale sensor networks. This paper introduces ELinFormer, which utilizes a novel Entropy-Aware Linear Attention (EALA) mechanism to achieve $O(N)$ complexity while maintaining state-of-the-art accuracy.

Methodology: Entropy-Aware Linear Attention (EALA)

The core innovation lies in approximating the standard softmax attention distribution by matching its entropy. The authors demonstrate that if two distributions share the same ranking and entropy, their structural similarity is high.

  1. Linear Entropy Approximation: Instead of computing the full $N \times N$ attention matrix, the authors use a first-order Taylor expansion to approximate the entropy. By leveraging matrix associativity, this calculation is reduced to $O(N)$ time.
  2. Entropy Equality: A linear function $\phi_\theta(x)$ is defined, and its parameter $\theta$ is analytically solved such that the entropy of the resulting linear distribution matches the approximated entropy of the softmax distribution.
  3. ELinFormer Architecture: This EALA module is integrated into a Transformer-based architecture, replacing standard attention to create a highly efficient forecasting model.

Experimental Setup

The model was evaluated on five benchmark highway traffic datasets from the Caltrans Performance Measurement System (PeMS): PEMS-BAY, PEMS03, PEMS04, PEMS07, and PEMS08. These datasets vary in size from 170 to 883 nodes, providing a robust test for scalability and accuracy.

Key Results and Discussion

ELinFormer demonstrates significant improvements in both computational efficiency and forecasting accuracy compared to the $O(N^2)$ baseline, STAEformer.

Efficiency and accuracy comparison of ELinFormer against the STAEformer baseline across different datasets.

As shown in the table above, ELinFormer achieves up to a 36.7% reduction in memory usage and an 18.8% reduction in training time on the PEMS03 dataset. Crucially, these efficiency gains do not come at the cost of performance; in many cases, ELinFormer outperforms the baseline in MAE and RMSE.

Comprehensive performance comparison across five PeMS datasets against various baseline models.

When compared against a broad spectrum of models—including RNN-based (DCRNN), TCN-based (Graph WaveNet), and other Transformer-based models (PDFormer)—ELinFormer consistently ranks among the top performers, often achieving the best or second-best results across all datasets.

Conclusion

The success of ELinFormer suggests that the specific non-linearity of softmax may not be the primary driver of attention's effectiveness in spatio-temporal modeling. Instead, maintaining a balanced weight distribution—quantified by entropy—is sufficient. By matching this entropy with a linear function, ELinFormer provides a scalable and powerful solution for large-scale multivariate time series forecasting. Future work includes developing custom CUDA kernels to further optimize hardware performance.

Summary updated

BibTeX
@misc{zhang2025efficient,
  author = {Mingtao Zhang and Guoli Yang and Zhanxing Zhu and Mengzhu Wang and Xiaoying Bai},
  title = {{Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Equality}},
  year = {2025},
  eprint = {2511.03190},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.03190},
}
Figure