Abstract
While Transformers have revolutionized natural language processing and computer vision, their application to Time Series Forecasting (TSF) has faced significant challenges. Recent research identifies a fundamental bottleneck: the tendency of Transformers to learn redundant, low-diversity latent representations in TSF tasks. This paper introduces the Sequence Complementor, a model-agnostic module that injects learnable sequences into the input stream to enrich feature representations. Combined with a novel diversification loss based on volume maximization, the Sequence Complementor significantly improves forecasting accuracy across 14 major benchmarks, establishing a new state-of-the-art in both long-term and short-term forecasting.
Introduction
Despite their architectural dominance in other domains, Transformers often struggle to outperform simple linear models in Time Series Forecasting (TSF). This performance gap has led to a critical investigation into the internal mechanics of Transformers when applied to temporal data. Researchers have discovered that the primary culprit is not the attention mechanism itself, but rather the "richness" of the internal representations the model generates. In TSF, Transformers frequently suffer from "thin" representations—latent features characterized by low entropy and low rank—which limit the model's ability to generalize across complex, non-stationary signals.
The Representation Bottleneck
The authors identify that time series data is often plagued by limited sample sizes and significant "domain shifts," where underlying patterns evolve over time. These factors cause standard Transformers to converge on redundant feature sets. By analyzing the relationship between latent space characteristics and model performance, the researchers found a near-linear correlation between the entropy of a model's representations and its forecasting error: higher entropy (indicating more diverse and informative features) consistently leads to lower Mean Squared Error (MSE).
Proposed Method: The Sequence Complementor
To address this bottleneck, the paper proposes the Sequence Complementor, a two-pronged architectural enhancement designed to boost information density within the model.
1. Learnable Sequences
The core of the proposal involves concatenating $K$ learnable sequences (vectors) to the original input patches. Unlike static padding or fixed positional encodings, these sequences are parameters optimized during the training process. They act as "information boosters," providing a set of learnable templates that the self-attention mechanism can query. This allows the model to pull in complementary context that may be missing from the raw input data, effectively expanding the model's "internal vocabulary."
2. Diversification Loss
To ensure that these $K$ sequences do not simply learn redundant information, the authors introduce a novel, differentiable diversification loss. This loss function utilizes volume maximization—calculated via singular value decomposition (SVD)—to force the learnable sequences to remain as orthogonal to one another as possible. By maximizing the "volume" spanned by these vectors, the model is incentivized to capture a wide array of distinct temporal features.
Theoretical and Empirical Insights
The Sequence Complementor is grounded in information theory. The authors provide mathematical proof that the addition of these complementors lowers the theoretical bound on the minimum mean-squared error (MMSE) by reducing conditional uncertainty within the latent space.
Furthermore, the module is designed to be model-agnostic. It can be integrated into existing high-performance architectures, such as iTransformer or PatchTST, without requiring fundamental changes to their core logic. This "plug-and-play" capability comes with minimal computational cost, typically resulting in only a 4-12% increase in training time.
Experimental Results
The effectiveness of the Sequence Complementor was validated across a massive gauntlet of 14 major TSF benchmarks:
- Long-term Forecasting: Testing on datasets including the ETT family, Electricity, Traffic, and Weather, the model secured the top performance spot in 47 out of the tested scenarios. It consistently outperformed recent state-of-the-art models like PatchTST.
- Short-term Forecasting: On the M4 competition dataset—comprising 100,000 different time series across various frequencies—the Sequence Complementor achieved first place in 14 out of 15 settings.
Qualitative analysis of the forecasts indicates that while vanilla Transformers often fail to capture sharp peaks or suffer from temporal drift, models equipped with Sequence Complementors track the ground truth with significantly higher fidelity.
Conclusion and Future Work
The Sequence Complementor represents a significant step forward in making Transformers more robust for time series tasks. By explicitly addressing the diversity of latent representations, the authors have provided a scalable method to overcome the inherent limitations of temporal data.
Future research directions include moving beyond the current "channel-independent" strategy to better capture cross-variate dependencies—how different variables in a multivariate system influence one another. Additionally, the authors suggest exploring "deformable patchifying" and more sophisticated normalization strategies to further refine the interaction between learnable sequences and real-world data.
Summary updated