Abstract
Time series forecasting in mission-critical domains requires models that are strictly causal, computationally efficient, and capable of multi-scale adaptation. Existing architectures often fail to satisfy all three criteria simultaneously—a challenge termed the "forecasting trilemma." This paper introduces AutoHFormer, a novel hierarchical autoregressive Transformer that addresses these gaps. By integrating segment-level parallel processing with step-wise refinement and a Dynamic Windowed Masked Attention (DWMA) mechanism, AutoHFormer achieves sub-quadratic complexity while maintaining strict temporal causality. Experimental results across eight benchmark datasets demonstrate that AutoHFormer provides state-of-the-art accuracy with up to a 10.76× training speedup and a 6.06× reduction in memory usage compared to leading models like PatchTST.
Introduction: The Time Series Forecasting Trilemma
Effective time series forecasting faces a fundamental "trilemma" involving three pillars:
- Strict Causality: Models must ensure that information from the future does not leak into the past during training, adhering to the autoregressive principle $p(x_t|x_{<t})$.
- Efficiency: Standard Transformers suffer from $O(L^2)$ complexity, while RNN-hybrids introduce sequential bottlenecks that hinder parallelization.
- Multi-scale Adaptability: Models must capture both short-term transients and long-term trends.
AutoHFormer is designed to resolve these conflicts by providing a parallelizable, causal solution that scales linearly with sequence length.
Methodology
The AutoHFormer architecture relies on three primary innovations to overcome the limitations of existing models.
Hierarchical Temporal Modeling
To mitigate the error propagation common in standard autoregressive models, AutoHFormer decomposes the forecasting task into a hierarchy. It first processes segment-level blocks in parallel to capture coarse trends. This is followed by an intra-segment sequential refinement process that handles fine-grained details. This dual-stage approach allows the model to maintain high accuracy across varying temporal scales.
Dynamic Windowed Masked Attention (DWMA)
To break the quadratic complexity of standard self-attention, AutoHFormer employs a sliding window approach. Unlike static windowing methods, DWMA utilizes a learnable exponential decay parameter ($\gamma$). This allows the model to adaptively determine the importance of recent versus distant history, maintaining $O(LW)$ complexity (where $W$ is the window size) while enforcing strict causality.
Adaptive Temporal Encoding
The model utilizes Precomputed Position Encodings (PPE) based on sinusoidal functions, stored in lookup tables to minimize computational overhead. When combined with the learnable decay rates in the attention mechanism, the model can adapt its temporal awareness to the specific frequency and characteristics of the input data.
Experimental Evaluation
The authors evaluated AutoHFormer against 11 state-of-the-art baselines, including PatchTST, iTransformer, and S-Mamba, across eight public benchmark datasets (ETT, PEMS04, PEMS08, Weather, and Electricity).
Key Results
- Performance: AutoHFormer achieved state-of-the-art results in 85% of experimental configurations (68 out of 80 cases).
- Efficiency: On the PEMS08 traffic dataset, the model was 10.76× faster to train and utilized 83.5% less GPU memory than PatchTST.
- Stability and Robustness: In long-horizon forecasting (up to 720 steps), AutoHFormer maintained high Pearson correlation scores (0.82–0.92). It also demonstrated superior resilience to noise, maintaining lower Mean Squared Error (MSE) than baselines under 15% additive noise conditions.
Discussion and Limitations
While AutoHFormer significantly improves efficiency and causality, it has limitations. Its reliance on local temporal context within its windows can lead to performance degradation if the proportion of missing data is excessively high. Furthermore, the model may occasionally overfit to local fluctuations in very stable regions of a time series. Future research will focus on developing more robust strategies for handling data gaps and further refining the balance between local and global pattern recognition.
Summary updated