Partial Correlation-Based Attention for Multivariate Time Series Forecasting

Won Kyung LeeAAAI 2020 · DOI

Abstract

Standard Transformer models for multivariate time series often fail to capture complex inter-variable dependencies due to the limitations of pairwise dot-product attention. This paper proposes a novel Partial Correlation-based Attention mechanism using ridge regression to model conditional dependencies. Combined with series-wise multi-resolution convolutional layers for domain-agnostic feature embedding, this architecture aims to improve forecasting accuracy across diverse continuous data domains such as traffic and finance.

Introduction

Predicting the future of multiple related variables—such as traffic flow across a city or stock prices within a sector—is a critical yet challenging task. Modern forecasting models often leverage Transformer architectures originally designed for Natural Language Processing (NLP). However, two major hurdles limit their effectiveness in the time series domain:

  1. Myopic Attention: Standard attention mechanisms calculate "dot-product" similarity between pairs of variables. In complex systems, the relationship between two variables is often mediated by a third. Standard attention is "myopic" because it ignores these conditional dependencies.
  2. The Embedding Gap: Transformers are optimized for discrete tokens. Time series data is continuous and often requires manual feature engineering (e.g., Fourier transforms) to be effectively embedded, requiring significant domain expertise.

Methodology

The proposed model introduces a modified Encoder-Decoder architecture designed to address these challenges through two primary innovations.

The proposed Encoder-Decoder architecture featuring series-wise convolutional layers and partial correlation-based attention.

Partial Correlation-based Self-Attention

Instead of simple dot-product similarity, the model calculates attention scores by fitting a multiple linear regression—specifically, Ridge Regression—of all "key" features against a "query" feature. By using the resulting regression coefficients ($\beta$) as attention scores, the model captures the partial correlation between variables. This approach effectively "controls" for the influence of other variables in the system, moving beyond simple pairwise matching to a more robust statistical modeling of dependencies.

Series-wise Multi-resolution Convolutional Layers

To bridge the embedding gap, the model employs data-driven convolutional layers with varying filter lengths. This multi-resolution approach is inspired by wavelet transformations but allows the model to learn optimal filters automatically from the data. This makes the architecture domain-agnostic, capable of extracting features from diverse fields like finance or transport without manual tuning.

Local Context and Temporal Patterns

Rather than treating single points in time as tokens, the model processes $l$-length subsequences. This allows the attention mechanism to consider local temporal patterns when calculating dependencies, providing a richer context for the forecasting task.

Proposed Evaluation

As this work represents a research proposal, quantitative results are pending. The author identifies two primary targets for benchmarking:

  • Traffic Forecasting: Utilizing real-world open datasets to evaluate spatial-temporal dependency capture.
  • Stock Market Prediction: Testing the model's predictive accuracy and stability in highly volatile financial environments.

Discussion and Key Insights

The shift from "similarity" to "partial correlation" represents a significant conceptual advancement in attention mechanisms. By treating attention as a statistical modeling problem, the model can better handle highly correlated "keys"—a common occurrence in multivariate data. The use of Ridge Regression ensures that attention weights remain stable and interpretable. Furthermore, the multi-scale learning enabled by convolutional layers allows the model to simultaneously capture short-term fluctuations and long-term trends.

Limitations and Future Work

While promising, the approach faces potential challenges in computational complexity, as calculating ridge regressions for every interaction may be more expensive than standard dot-products. Future work will focus on rigorous benchmarking against state-of-the-art models (LSTMs, Graph Neural Networks) and exploring the incorporation of prior knowledge, such as physical road network layouts, to further guide dependency discovery.

Conclusion

This paper outlines a novel framework for multivariate time series forecasting that replaces heuristic similarity measures with statistically grounded partial correlations. By automating feature extraction through multi-resolution convolutions, the proposed architecture offers a robust, domain-agnostic solution for complex forecasting tasks.

Summary updated

BibTeX
@inproceedings{lee2020partial,
  author = {Won Kyung Lee},
  title = {{Partial Correlation-Based Attention for Multivariate Time Series Forecasting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7132},
}
Figure