Sequence Complementor: Complementing Transformers for Time Series Forecasting with Learnable Sequences

Xiwen Chen, Peijie Qiu, Wenhui Zhu, Huayu Li, Hao Wang, Aristeidis Sotiras, Yalin Wang, Abolfazl Razi

Abstract

While Transformers have revolutionized natural language processing and computer vision, their application to Time Series Forecasting (TSF) has faced significant challenges. Recent research identifies a fundamental bottleneck: the tendency of Transformers to learn redundant, low-diversity latent representations in TSF tasks. This paper introduces the Sequence Complementor, a model-agnostic module that injects learnable sequences into the input stream to enrich feature representations. Combined with a novel diversification loss based on volume maximization, the Sequence Complementor significantly improves forecasting accuracy across 14 major benchmarks, establishing a new state-of-the-art in both long-term and short-term forecasting.

Introduction

Despite their architectural dominance in other domains, Transformers often struggle to outperform simple linear models in Time Series Forecasting (TSF). This performance gap has led to a critical investigation into the internal mechanics of Transformers when applied to temporal data. Researchers have discovered that the primary culprit is not the attention mechanism itself, but rather the "richness" of the internal representations the model generates. In TSF, Transformers frequently suffer from "thin" representations—latent features characterized by low entropy and low rank—which limit the model's ability to generalize across complex, non-stationary signals.

The Representation Bottleneck

The authors identify that time series data is often plagued by limited sample sizes and significant "domain shifts," where underlying patterns evolve over time. These factors cause standard Transformers to converge on redundant feature sets. By analyzing the relationship between latent space characteristics and model performance, the researchers found a near-linear correlation between the entropy of a model's representations and its forecasting error: higher entropy (indicating more diverse and informative features) consistently leads to lower Mean Squared Error (MSE).

Proposed Method: The Sequence Complementor

To address this bottleneck, the paper proposes the Sequence Complementor, a two-pronged architectural enhancement designed to boost information density within the model.

1. Learnable Sequences

The core of the proposal involves concatenating $K$ learnable sequences (vectors) to the original input patches. Unlike static padding or fixed positional encodings, these sequences are parameters optimized during the training process. They act as "information boosters," providing a set of learnable templates that the self-attention mechanism can query. This allows the model to pull in complementary context that may be missing from the raw input data, effectively expanding the model's "internal vocabulary."

2. Diversification Loss

To ensure that these $K$ sequences do not simply learn redundant information, the authors introduce a novel, differentiable diversification loss. This loss function utilizes volume maximization—calculated via singular value decomposition (SVD)—to force the learnable sequences to remain as orthogonal to one another as possible. By maximizing the "volume" spanned by these vectors, the model is incentivized to capture a wide array of distinct temporal features.

Theoretical and Empirical Insights

The Sequence Complementor is grounded in information theory. The authors provide mathematical proof that the addition of these complementors lowers the theoretical bound on the minimum mean-squared error (MMSE) by reducing conditional uncertainty within the latent space.

Furthermore, the module is designed to be model-agnostic. It can be integrated into existing high-performance architectures, such as iTransformer or PatchTST, without requiring fundamental changes to their core logic. This "plug-and-play" capability comes with minimal computational cost, typically resulting in only a 4-12% increase in training time.

Experimental Results

The effectiveness of the Sequence Complementor was validated across a massive gauntlet of 14 major TSF benchmarks:

  • Long-term Forecasting: Testing on datasets including the ETT family, Electricity, Traffic, and Weather, the model secured the top performance spot in 47 out of the tested scenarios. It consistently outperformed recent state-of-the-art models like PatchTST.
  • Short-term Forecasting: On the M4 competition dataset—comprising 100,000 different time series across various frequencies—the Sequence Complementor achieved first place in 14 out of 15 settings.

Qualitative analysis of the forecasts indicates that while vanilla Transformers often fail to capture sharp peaks or suffer from temporal drift, models equipped with Sequence Complementors track the ground truth with significantly higher fidelity.

Conclusion and Future Work

The Sequence Complementor represents a significant step forward in making Transformers more robust for time series tasks. By explicitly addressing the diversity of latent representations, the authors have provided a scalable method to overcome the inherent limitations of temporal data.

Future research directions include moving beyond the current "channel-independent" strategy to better capture cross-variate dependencies—how different variables in a multivariate system influence one another. Additionally, the authors suggest exploring "deformable patchifying" and more sophisticated normalization strategies to further refine the interaction between learnable sequences and real-world data.

Summary updated

Abstract

Since its introduction, the transformer has shifted the development trajectory away from traditional models (e.g., RNN, MLP) in time series forecasting, which is attributed to its ability to capture global dependencies within temporal tokens. Follow-up studies have largely involved altering the tokenization and self-attention modules to better adapt Transformers for addressing special challenges like non-stationarity, channel-wise dependency, and variable correlation in time series. However, we found that the expressive capability of sequence representation is a key factor influencing Transformer performance in time forecasting after investigating several representative methods, where there is an almost linear relationship between sequence representation entropy and mean square error, with more diverse representations performing better. In this paper, we propose a novel attention mechanism with Sequence Complementors and prove feasible from an information theory perspective, where these learnable sequences are able to provide complementary information beyond current input to feed attention. We further enhance the Sequence Complementors via a diversification loss that is theoretically covered. The empirical evaluation of both long-term and short-term forecasting has confirmed its superiority over the recent state-of-the-art methods.
BibTeX
@inproceedings{chen2025sequence,
  author = {Xiwen Chen and Peijie Qiu and Wenhui Zhu and Huayu Li and Hao Wang and Aristeidis Sotiras and Yalin Wang and Abolfazl Razi},
  title = {{Sequence Complementor: Complementing Transformers for Time Series Forecasting with Learnable Sequences}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i15.33747},
}
Figure