AutoHFormer: Efficient Hierarchical Autoregressive Transformer for Time Series Prediction

Qianru Zhang, Honggang Wen, Ming Li, Dong Huang, Siu-Ming Yiu, Christian S. Jensen, Pietro Liò

Abstract

Time series forecasting in mission-critical domains requires models that are strictly causal, computationally efficient, and capable of multi-scale adaptation. Existing architectures often fail to satisfy all three criteria simultaneously—a challenge termed the "forecasting trilemma." This paper introduces AutoHFormer, a novel hierarchical autoregressive Transformer that addresses these gaps. By integrating segment-level parallel processing with step-wise refinement and a Dynamic Windowed Masked Attention (DWMA) mechanism, AutoHFormer achieves sub-quadratic complexity while maintaining strict temporal causality. Experimental results across eight benchmark datasets demonstrate that AutoHFormer provides state-of-the-art accuracy with up to a 10.76× training speedup and a 6.06× reduction in memory usage compared to leading models like PatchTST.

Introduction: The Time Series Forecasting Trilemma

Effective time series forecasting faces a fundamental "trilemma" involving three pillars:

  1. Strict Causality: Models must ensure that information from the future does not leak into the past during training, adhering to the autoregressive principle $p(x_t|x_{<t})$.
  2. Efficiency: Standard Transformers suffer from $O(L^2)$ complexity, while RNN-hybrids introduce sequential bottlenecks that hinder parallelization.
  3. Multi-scale Adaptability: Models must capture both short-term transients and long-term trends.
Comparison of architectural approaches to time series forecasting.

AutoHFormer is designed to resolve these conflicts by providing a parallelizable, causal solution that scales linearly with sequence length.

Methodology

The AutoHFormer architecture relies on three primary innovations to overcome the limitations of existing models.

Hierarchical Temporal Modeling

To mitigate the error propagation common in standard autoregressive models, AutoHFormer decomposes the forecasting task into a hierarchy. It first processes segment-level blocks in parallel to capture coarse trends. This is followed by an intra-segment sequential refinement process that handles fine-grained details. This dual-stage approach allows the model to maintain high accuracy across varying temporal scales.

Dynamic Windowed Masked Attention (DWMA)

To break the quadratic complexity of standard self-attention, AutoHFormer employs a sliding window approach. Unlike static windowing methods, DWMA utilizes a learnable exponential decay parameter ($\gamma$). This allows the model to adaptively determine the importance of recent versus distant history, maintaining $O(LW)$ complexity (where $W$ is the window size) while enforcing strict causality.

The proposed sliding window attention mechanism with learnable exponential decay.

Adaptive Temporal Encoding

The model utilizes Precomputed Position Encodings (PPE) based on sinusoidal functions, stored in lookup tables to minimize computational overhead. When combined with the learnable decay rates in the attention mechanism, the model can adapt its temporal awareness to the specific frequency and characteristics of the input data.

Experimental Evaluation

The authors evaluated AutoHFormer against 11 state-of-the-art baselines, including PatchTST, iTransformer, and S-Mamba, across eight public benchmark datasets (ETT, PEMS04, PEMS08, Weather, and Electricity).

Key Results

  • Performance: AutoHFormer achieved state-of-the-art results in 85% of experimental configurations (68 out of 80 cases).
  • Efficiency: On the PEMS08 traffic dataset, the model was 10.76× faster to train and utilized 83.5% less GPU memory than PatchTST.
  • Stability and Robustness: In long-horizon forecasting (up to 720 steps), AutoHFormer maintained high Pearson correlation scores (0.82–0.92). It also demonstrated superior resilience to noise, maintaining lower Mean Squared Error (MSE) than baselines under 15% additive noise conditions.

Discussion and Limitations

While AutoHFormer significantly improves efficiency and causality, it has limitations. Its reliance on local temporal context within its windows can lead to performance degradation if the proportion of missing data is excessively high. Furthermore, the model may occasionally overfit to local fluctuations in very stable regions of a time series. Future research will focus on developing more robust strategies for handling data gaps and further refining the balance between local and global pattern recognition.

Summary updated

Abstract

Time series forecasting requires architectures that simultaneously achieve three competing objectives: (1) strict temporal causality for reliable predictions, (2) sub-quadratic complexity for practical scalability, and (3) multi-scale pattern recognition for accurate long-horizon forecasting. We introduce AutoHFormer, a hierarchical autoregressive transformer that addresses these challenges through three key innovations: 1) Hierarchical Temporal Modeling: Our architecture decomposes predictions into segment-level blocks processed in parallel, followed by intra-segment sequential refinement. This dual-scale approach maintains temporal coherence while enabling efficient computation. 2) Dynamic Windowed Attention: The attention mechanism employs learnable causal windows with exponential decay, reducing complexity while preserving precise temporal relationships. This design avoids both the anti-causal violations of standard transformers and the sequential bottlenecks of RNN hybrids. 3) Adaptive Temporal Encoding: a novel position encoding system is adopted to capture time patterns at multiple scales. It combines fixed oscillating patterns for short-term variations with learnable decay rates for long-term trends. Comprehensive experiments demonstrate that AutoHFormer 10.76X faster training and 6.06X memory reduction compared to PatchTST on PEMS08, while maintaining consistent accuracy across 96-720 step horizons in most of cases. These breakthroughs establish new benchmarks for efficient and precise time series modeling. Implementations of our method and all baselines in hierarchical autoregressive mechanism are available at https://github.com/lizzyhku/Autotime.
BibTeX
@inproceedings{zhang2026autohformer,
  author = {Qianru Zhang and Honggang Wen and Ming Li and Dong Huang and Siu-Ming Yiu and Christian S. Jensen and Pietro Liò},
  title = {{AutoHFormer: Efficient Hierarchical Autoregressive Transformer for Time Series Prediction}},
  booktitle = {ICDE},
  year = {2026},
  doi = {10.1109/icde65706.2026.00021},
}
Figure