Fewer yet critical: Reducing Redundant Token Dependencies for Transformer-based Time Series Forecasting
Lists
Only you see this
Abstract
Time series forecasting (TSF) is important in real-world applications. Recently, Transformer-based methods have achieved strong performance by modeling token dependencies through attention mechanisms. However, existing methods are usually trained mainly with prediction error losses, which may cause models to exploit both critical and redundant token dependencies. Such redundant dependencies can introduce irrelevant information and weaken generalization. To address this issue, we propose a simple yet effective token dependency selection strategy. Specifically, by jointly introducing the attention entropy constraint and prediction error constraint, the model can identify fewer but more critical inter-token dependencies and perform forecasting based on them, thus avoiding the interference of redundant dependencies. The proposed method can be easily extended to various Transformer-based TSF models. Experiments on multiple TSF datasets demonstrate its effectiveness.
BibTeX
@misc{zhang2025fewer,
author = {Jianqi Zhang and Yuchan Liu and Zeen Song and Yuefei Li and Fanjiang Xu},
title = {{Fewer yet critical: Reducing Redundant Token Dependencies for Transformer-based Time Series Forecasting}},
year = {2025},
eprint = {2503.06867},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2503.06867},
}Loading related content...