NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting

Kai Chen, Guang Chen, Dan Xu, Lijun Zhang, Yuyao Huang, Alois C. KnollarXiv 2021 · OpenReview

TL;DR

This paper introduces NAST, a novel Non-Autoregressive Spatial-Temporal Transformer for time series forecasting. It tackles the issues of cumulative errors and inefficient processing of spatial-temporal dependencies in existing Transformer models, achieving state-of-the-art performance with improved real-time forecasting.

What Problem Does This Paper Address?

Transformers have revolutionized Natural Language Processing (NLP), but their application to time series forecasting has been limited. Canonical Transformer models often use an autoregressive decoding approach, which can lead to a compounding of prediction errors over time. Furthermore, effectively capturing the intricate spatial and temporal dependencies present in many time series datasets (like predicting future object locations from sensor data) remains a significant challenge for Transformer architectures. Existing methods either process spatial and temporal information separately or in a stacked, sequential manner, failing to integrate them effectively for joint modeling.

Methods

The core innovation of NAST lies in two key components:

  • Non-Autoregressive (Non-AR) Decoding with Query Generation Block (QGB): Instead of predicting one step at a time, NAST generates all future queries simultaneously in a single step using a novel QGB. This parallel processing significantly reduces decoding time and, crucially, avoids the accumulation of errors inherent in autoregressive methods.
  • Novel Spatial-Temporal Attention Mechanism: This mechanism integrates spatial and temporal dependencies by first predicting separate spatial and temporal attention maps. A "temporal influence map" is then derived from the temporal attention map and combined with the spatial attention map. This creates a unified spatial-temporal attention map, allowing the model to jointly process these intertwined dependencies.

Datasets

The proposed model was evaluated on two datasets:

  • NuScenes: A large-scale autonomous driving dataset providing scene data and object bounding box tracking.
  • SMARTS-EGO: A newly generated ego-centric future localization dataset created using the SMARTS simulator, designed to provide longer sequences than NuScenes for more robust evaluation.

Key Insights and Ideas

  • Non-Autoregression is Key for Long-Term Forecasting: The paper strongly demonstrates that autoregressive decoding, while common, becomes a bottleneck for long time series due to error accumulation. Non-autoregressive decoding, especially with parallel query generation, offers a significant advantage.
  • Integrated Spatial-Temporal Attention is Superior: Treating spatial and temporal dependencies separately or in a simple stacked manner is less effective than a truly integrated approach. The proposed "temporal influence map" is a clever way to bridge these two aspects.
  • Transformer's Potential in Time Series is Underexplored: This work highlights that Transformers, with appropriate architectural modifications, can outperform traditional RNN-based methods in time series forecasting, especially for complex spatio-temporal tasks.

Key Results

  • NAST achieved state-of-the-art performance on both the SMARTS-EGO and NuScenes datasets, outperforming existing RNN-based and Transformer-based models.
  • The non-autoregressive decoding significantly reduced decoding time compared to autoregressive methods.
  • Ablation studies confirmed the effectiveness of both the non-autoregressive decoding and the novel spatial-temporal attention mechanism, showing that the integrated approach outperformed parallel or stacked attention strategies.

Limitations and Future Work

The authors acknowledge that while their spatial-temporal attention mechanism is effective, further research into more explainable methods for generating queries and modeling these complex attentions is needed. They also suggest that NAST can serve as a strong baseline for future Transformer-based time series forecasting research.

Summary updated

Abstract

Although Transformer has made breakthrough success in widespread domains especially in Natural Language Processing (NLP), applying it to time series forecasting is still a great challenge. In time series forecasting, the autoregressive decoding of canonical Transformer models could introduce huge accumulative errors inevitably. Besides, utilizing Transformer to deal with spatial-temporal dependencies in the problem still faces tough difficulties.~To tackle these limitations, this work is the first attempt to propose a Non-Autoregressive Transformer architecture for time series forecasting, aiming at overcoming the time delay and accumulative error issues in the canonical Transformer. Moreover, we present a novel spatial-temporal attention mechanism, building a bridge by a learned temporal influence map to fill the gaps between the spatial and temporal attention, so that spatial and temporal dependencies can be processed integrally. Empirically, we evaluate our model on diversified ego-centric future localization datasets and demonstrate state-of-the-art performance on both real-time and accuracy.
BibTeX
@misc{chen2021nast,
  author = {Kai Chen and Guang Chen and Dan Xu and Lijun Zhang and Yuyao Huang and Alois C. Knoll},
  title = {{NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting}},
  year = {2021},
  eprint = {2102.05624},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.05624},
}
Figure