Enhancing Time Series Forecasting via Logic-Inspired Regularization

Jianqi Zhang, Jingyao Wang, Xingchen Shen, Wenwen Qiang

Abstract

Time series forecasting (TSF) plays a crucial role in many applications. Transformer-based methods are one of the mainstream techniques for TSF. Existing methods treat all token dependencies equally. However, we find that the effectiveness of token dependencies varies across different forecasting scenarios, and existing methods ignore these differences, which affects their performance. This raises two issues: (1) What are effective token dependencies? (2) How can we learn effective dependencies? From a logical perspective, we align Transformer-based TSF methods with the logical framework and define effective token dependencies as those that ensure the tokens as atomic formulas. We then align the learning process of Transformer methods with the process of obtaining atomic formulas in logic, which inspires us to design a method for learning these effective dependencies. Specifically, we propose Attention Logic Regularization (Attn-L-Reg), a plug-and-play method that guides the model to use fewer but more effective dependencies by making the attention map sparse, thereby ensuring the tokens as atomic formulas and improving prediction performance. Extensive experiments and theoretical analysis confirm the effectiveness of Attn-L-Reg.

Introduction

In time series forecasting (TSF), Transformers have become a dominant architecture due to their self-attention mechanism's ability to capture long-range dependencies. However, standard Transformers suffer from a critical flaw: Equal Dependency Handling. These models assume every "token" (a segment of the time series) should potentially attend to every other token.

Empirical analysis suggests that many of these dependencies are actually noise. For instance, when predicting immediate future points that do not span a full cycle, dependencies related to long-term periodic patterns might be redundant or even detrimental to performance. The challenge lies in identifying which dependencies are effective and which are redundant without excessive manual tuning.

Methodology: Attention Logic Regularization (Attn-L-Reg)

To address the redundancy in attention mechanisms, this research turns to formal logic. By bridging the gap between deep learning and symbolic logic, tokens are treated as "atomic formulas"—the simplest, indivisible units of meaning.

  1. Logical Alignment: An "effective" dependency is defined as one that helps form these atomic formulas. If a token's representation can be simplified without losing predictive power, the original representation was "composite" (redundant) rather than "atomic."
  2. Attn-L-Reg Mechanism: To enforce this atomicity, the authors propose a plug-and-play regularization term. It applies L1 regularization directly to the attention maps ($\Xi$). This encourages the model to be "minimalist," utilizing the fewest possible dependencies to achieve optimal prediction.
  3. Objective Function: The model minimizes a combined loss function consisting of the standard prediction error (MSE) and the weighted sum of the L1 norms of the attention maps across all encoder layers: $\ell_{pred} + \sum_{i=1}^{L} \alpha_i \ell_{reg}(\Xi_i)$.
The overall framework of Attn-L-Reg integrated into a Transformer-based encoder.

Experimental Results

The proposed method was evaluated on six widely-used real-world TSF benchmarks, including ECL (Electricity), Traffic, Weather, ETTh2, Solar-Energy, and PEMS03. Attn-L-Reg was integrated into two state-of-the-art models: iTransformer and PatchTST.

Performance Gains

Across nearly all datasets and look-back windows, the logic-regularized versions consistently outperformed the original models. For example, on the high-dimensional Traffic dataset (862 variables), iTransformer+Attn-L-Reg achieved a significant boost by successfully pruning redundant dependencies.

Table 1: Comprehensive performance comparison of Attn-L-Reg against state-of-the-art TSF models.

Sparsity and Accuracy

On the ECL dataset, the regularized model increased attention map sparsity from 1.2% to 54.0% while simultaneously reducing the MSE from 0.148 to 0.133. This demonstrates that "less is more" in the context of attention for time series; by using logic as a sparsity prior, the model learns more meaningful, indivisible features rather than overfitting to noise.

Conclusion

Attn-L-Reg provides a theoretically grounded and practically effective method for improving Transformer-based TSF. By treating tokens as logical atomic formulas and applying L1 regularization to attention maps, the model achieves better generalization and higher accuracy. Future work may involve automating hyperparameter selection for the regularization strength and extending these logic-inspired constraints to non-Transformer architectures like State Space Models.

Summary updated

Abstract

Time series forecasting (TSF) plays a crucial role in many applications. Transformer-based methods are one of the mainstream techniques for TSF. Existing methods treat all token dependencies equally. However, we find that the effectiveness of token dependencies varies across different forecasting scenarios, and existing methods ignore these differences, which affects their performance. This raises two issues: (1) What are effective token dependencies? (2) How can we learn effective dependencies? From a logical perspective, we align Transformer-based TSF methods with the logical framework and define effective token dependencies as those that ensure the tokens as atomic formulas (Issue 1). We then align the learning process of Transformer methods with the process of obtaining atomic formulas in logic, which inspires us to design a method for learning these effective dependencies (Issue 2). Specifically, we propose Attention Logic Regularization (Attn-L-Reg), a plug-and-play method that guides the model to use fewer but more effective dependencies by making the attention map sparse, thereby ensuring the tokens as atomic formulas and improving prediction performance. Extensive experiments and theoretical analysis confirm the effectiveness of Attn-L-Reg.
BibTeX
@misc{zhang2025enhancing,
  author = {Jianqi Zhang and Jingyao Wang and Xingchen Shen and Wenwen Qiang},
  title = {{Enhancing Time Series Forecasting via Logic-Inspired Regularization}},
  year = {2025},
  eprint = {2503.06867},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.06867},
}
Figure