Temporal Dependencies and Spatio-Temporal Patterns of Time Series Models

Md. Khairul IslamAAAI 2024 · DOI

Abstract

Deep learning models have significantly advanced time series forecasting, yet their "black-box" nature limits their utility in critical domains like healthcare and finance. This paper presents a three-phase framework designed to enhance the interpretability and trustworthiness of these models. By transitioning from model-specific attention mechanisms to a novel "Windowed Temporal Saliency Rescaling" technique, we uncover how models capture complex spatio-temporal dependencies, specifically within the context of pandemic forecasting.

Introduction

The integration of deep learning into time series forecasting has revolutionized fields requiring high-precision temporal predictions. However, the inherent complexity of these models often obscures the underlying logic of their outputs. Unlike image or text data, time series data involves shifting temporal dependencies where the significance of a feature is contingent upon its timing. Current interpretability methods often struggle with model specificity—limiting their application to specific architectures like Transformers—and recency bias, which overemphasizes the most recent data points at the expense of lagged effects.

Problem Statement

The primary challenge in interpreting time series models lies in the "black-box" nature of deep learning architectures. Existing methods frequently fail to capture the nuances of spatio-temporal data for two main reasons:

  1. Architectural Constraints: Many interpretation tools are tied to specific model internals (e.g., attention weights), making them non-transferable.
  2. Temporal Myopia: Standard saliency methods often assume that the most recent inputs are the most influential, ignoring historical events that may have delayed impacts on future outcomes.

Methodology

This research proposes a structured three-phase approach to address these interpretability gaps.

Phase I: Spatio-Temporal Pattern Analysis

In the initial phase, we utilize the Temporal Fusion Transformer (TFT) to predict COVID-19 infection rates. By extracting the model’s internal self-attention weights, we can visualize the geographical areas and specific time steps the model prioritizes. This allows for a granular understanding of how the model processes spatial data across the United States.

Spatial distribution of model attention across US counties, highlighting regional priorities in pandemic forecasting.

Phase II: Feature Interaction and Sensitivity Analysis

To move beyond model-specific internals, we employ Sensitivity Analysis. This "black-box" method quantifies the impact of input feature variations—such as infection rates across different age demographics—on the final model output. This phase validates whether the model's learned patterns align with real-world epidemiological data.

Phase III: Windowed Temporal Saliency Rescaling

The core contribution of this work is the "Windowed Temporal Saliency Rescaling" algorithm. This method calculates the importance of each input time step independently and rescales them across the entire "look-back" window. This approach ensures that significant shifts occurring early in the observation period are not overshadowed by more recent, but potentially less impactful, data.

Results and Discussion

The Temporal Fusion Transformer (TFT) demonstrated superior performance, outperforming four state-of-the-art deep learning models across five regression metrics. Key insights include:

  • Attention as a Proxy for Reality: Self-attention weights were found to be highly correlated with raw data patterns and case reporting behaviors, suggesting that "white-box" internals can effectively mirror real-world dynamics.
  • Capturing the "Lag": The research highlighted that critical feature shifts often have delayed effects. Our proposed windowed method successfully identifies these dynamics where standard saliency methods fail.
  • Demographic Validation: Sensitivity rankings for different age groups showed high alignment with CDC ground-truth data, confirming that black-box interpretation can be as reliable as internal weight analysis.

Conclusion and Future Work

While Phase I provided deep insights into model internals, its specificity to the TFT architecture remains a limitation. Phases II and III address this by developing more generalized, black-box interpretation techniques. As shown in the project status table, the framework is nearing completion, with the novel saliency rescaling algorithm currently in the final benchmarking stage.

Current progress status of the three-phase research framework.

Future work will focus on finalizing the Windowed Temporal Saliency Rescaling algorithm and testing its generalizability across diverse classification and regression datasets. The ultimate goal is to provide a more "honest" and robust view of feature importance in time series forecasting.

Summary updated

BibTeX
@inproceedings{islam2024temporal,
  author = {Md. Khairul Islam},
  title = {{Temporal Dependencies and Spatio-Temporal Patterns of Time Series Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30396},
}
Figure