Abstract
Despite the notable advancements in numerous Transformer-based models, the task of long multi-horizon time series forecasting remains a persistent challenge, especially towards explainability. Focusing on commonly used saliency maps in explaining DNN in general, our quest is to build attention-based architecture that can automatically encode saliency-related temporal patterns by establishing connections with appropriate attention heads. Hence, this paper introduces Temporal Saliency Detection (TSD), an effective approach that builds upon the attention mechanism and applies it to multi-horizon time series prediction. While our proposed architecture adheres to the general encoder-decoder structure, it undergoes a significant renovation in the encoder component, wherein we incorporate a series of information contracting and expanding blocks inspired by the U-Net style architecture. The TSD approach facilitates the multiresolution analysis of saliency patterns by condensing multi-heads, thereby progressively enhancing the forecasting of complex time series data. Empirical evaluations illustrate the superiority of our proposed approach compared to other models across multiple standard benchmark datasets in diverse far-horizon forecasting settings. The initial TSD achieves substantial relative improvements of 31% and 46% over several models in the context of multivariate and univariate prediction.
Introduction
Long-term, multi-horizon time series forecasting—predicting many steps into the future—is a critical yet notoriously difficult task. While Transformer-based models have revolutionized the field, they face two major hurdles: computational complexity and explainability. Standard self-attention scales quadratically ($O(L^2)$) with sequence length, making long-term predictions computationally expensive. Furthermore, most "sparse" Transformer variants (like Informer) reduce complexity by ignoring certain data points, which can lead to information loss and make it harder to understand the underlying rationale behind a model's prediction.
This research argues that effective forecasting requires models that do not simply "thin out" data, but instead automatically identify temporal saliency patterns. These are the specific segments of a timeline that are most influential for future outcomes across different time scales. By establishing connections with appropriate attention heads, the proposed Temporal Saliency Detection (TSD) model aims to bridge the gap between high-performance forecasting and model interpretability.
Methodology: The TSD Architecture
The core innovation of this work is the Temporal Saliency Detection (TSD) block, which renovates the standard Transformer encoder-decoder structure with a design inspired by U-Net, a popular architecture in medical image segmentation.
U-Net Style Renovation
The encoder uses a series of "contracting" paths (downsampling via MaxPool) to capture high-level, coarse temporal patterns, followed by "expanding" paths (upsampling via ConvTranspose1d) to refine those patterns back to the original resolution. This multi-resolution analysis allows the model to treat time series forecasting similarly to a semantic segmentation task, identifying salient patterns at various scales.
Skip Connections and Temporal Self-Attention (TSA)
Much like the original U-Net, TSD uses skip connections to link the contracting and expanding paths. This ensures that fine-grained local information is preserved during the compression phase. Before passing data into these multi-resolution blocks, the model utilizes a single, efficient Temporal Self-Attention (TSA) block to encode global correlations. Unlike models that rely on deep stacks of encoders, TSD demonstrates that a single, well-architected encoder-decoder pair can be more effective for multi-horizon tasks.
Experimental Setup
The model was evaluated on several standard "far-horizon" benchmarks representing diverse challenges:
- ETT (Electricity Transformer Temperature): Real-world data for electric power deployment (ETTh1, ETTh2, ETTm1), characterized by clear seasonality.
- ILI (Influenza-Like Illness): Weekly recorded influenza patient ratios from the CDC, featuring high fluctuation.
- Exchange: Daily currency exchange rates from 1990 to 2016, representing complex, non-stationary trends.
Results and Discussion
TSD demonstrated significant performance gains across all tested horizons and datasets. As shown in the performance comparison, the TSD model (represented by the red line) consistently maintains the lowest error rates compared to established baselines.
Key Performance Metrics
- Multivariate Superiority: TSD outperformed six major baselines (including Autoformer and Reformer) in nearly all tested horizons, achieving an average 31% reduction in Mean Squared Error (MSE) across 46 reported results.
- Univariate Gains: In univariate settings, the improvement was even more pronounced, with a 46% relative error reduction.
- Dataset-Specific Success: On the ETTh1 and ETTm1 datasets, TSD achieved a staggering 65% reduction in MSE compared to previous state-of-the-art models.
Insights on Saliency
Ablation studies revealed that Max Pooling is more stable than Average Pooling for long-term forecasting. This suggests that "peak" values in a time segment are often more salient than average behavior when looking far into the future. Furthermore, TSD achieves efficiency through intelligent design—condensing information into lower-resolution embeddings—rather than through data sparsification.
Conclusion
The Temporal Saliency Detection approach provides a robust framework for explainable time series forecasting. While the model shows superior performance, it currently faces an optimization bottleneck; finding the optimal configuration required extensive manual trial-and-error iterations. Future work will focus on automating this hyperparameter search and exploring more advanced U-Net derivatives, such as those used in latent diffusion models, to further enhance the extraction of complex temporal features.
Summary updated