TL;DR
The Dozerformer introduces a novel "Dozer Attention" mechanism that breaks the quadratic complexity bottleneck of standard Transformers in time series forecasting. By combining local, strided, and horizon-adaptive (Vary) attention components, it achieves state-of-the-art accuracy across nine benchmarks while significantly reducing computational overhead and memory usage.
What Problem Does This Paper Address?
Transformers have revolutionized multivariate time series (MTS) forecasting by capturing long-term dependencies better than older RNN or CNN architectures. However, they face two major hurdles:
- Quadratic Complexity: The standard self-attention mechanism scales quadratically ($O(L^2)$) with sequence length, making it expensive to process long historical look-back windows.
- Redundant History: Standard models use the entire historical sequence to predict every future time step. In reality, predicting what happens five minutes from now usually requires a different historical context than predicting what happens five days from now. Using the full history for every prediction point is not only inefficient but can introduce noise that degrades accuracy.
Methods
The authors propose the Dozerformer framework, which decomposes time series into seasonal and trend components. While a simple linear layer handles the trend, the seasonal component is processed by the Dozer Attention mechanism, which consists of three sparse sub-components:
- Local Component: Each query only attends to keys within a small, localized window. This captures immediate temporal correlations (e.g., what happened an hour ago is highly relevant to right now).
- Stride Component: Queries attend to keys at predefined intervals (strides). This is specifically designed to capture seasonality, such as daily or weekly patterns, without checking every single intervening time step.
- Vary Component: This is the most novel part. It dynamically adjusts the length of the historical sequence used based on the forecasting horizon. For near-term predictions, it uses a shorter "look-back"; as the model predicts further into the future, the window of attended history expands.
By combining these, the model ignores "gray areas" in the attention matrix—redundant query-key pairs that don't contribute to prediction accuracy—thereby achieving linear complexity relative to the input length.
Datasets
The model was evaluated on nine widely recognized public MTS benchmarks:
- ETT (Electricity Transformer Temperature): Four subsets (ETTh1, ETTh2, ETTm1, ETTm2).
- Traffic: Hourly road occupancy rates.
- Electricity: Hourly consumption of 321 clients.
- Weather: Meteorological indicators recorded every 10 minutes.
- Exchange-Rate: Daily exchange rates for eight countries.
- ILI (Influenza-Like Illness): Weekly patient data (notable for its smaller size and unique horizons).
Key Insights and Ideas
- Sparsity is a Feature, Not a Bug: The researchers found that excluding a large subset of historical time steps doesn't just save memory—it actually improves accuracy. By forcing the model to focus on local, seasonal, and horizon-relevant data, they effectively "denoise" the input.
- Horizon-Adaptive Attention: The "Vary" component acknowledges that the "receptive field" of a forecast should change. This mimics how humans think: you don't need a year of data to guess if it will rain in ten minutes, but you might need it to guess the weather next month.
- Efficiency Gains: Dozerformer manages to outperform models like PatchTST while using significantly fewer parameters and floating-point operations (FLOPs). For example, on the ETTh1 dataset, Dozerformer has ~6x fewer parameters and ~17x fewer FLOPs than PatchTST.
Key Results
- Superior Accuracy: Dozerformer achieved the best performance in 48 out of 72 test cases across the nine datasets.
- MSE Reductions: Compared to other state-of-the-art models, it reduced Mean Squared Error (MSE) by 0.4% over PatchTST, 8.3% over DLinear, and a massive 40.6% over Crossformer.
- Computational Efficiency: The Dozer self-attention mechanism reduced the number of query-key pairs to roughly 31.7% (encoder) and 28.4% (decoder) of what a full-attention Transformer would require.
Limitations and Future Work
- Hyperparameter Sensitivity: The optimal sizes for the Local ($w$), Stride ($s$), and Vary ($v$) components vary by dataset. Finding the "sweet spot" for these requires manual tuning or grid searching, which can be time-consuming.
- Multi-Scale Dependencies: While the model handles seasonality well via the Stride component, the authors suggest that future versions could be modified to better model complex, multi-scale temporal dependencies (e.g., overlapping daily, weekly, and monthly cycles).
- Adaptive Selection: The authors aim to move toward a version of the model that can automatically and adaptively select the best subsets of historical time steps without as much manual configuration.
Summary updated