VCformer: Variable Correlation Transformer with Inherent Lagged Correlation for Multivariate Time Series Forecasting

Yingnan Yang, Qingling Zhu, Jianyong Chen

Introduction

Multivariate time series (MTS) forecasting is critical for applications ranging from urban traffic management to power grid optimization. A fundamental challenge in MTS is that variables rarely exhibit perfectly synchronized behavior; instead, changes in one variable often influence others after a specific time delay, known as a "lag." Traditional Transformer architectures typically rely on vanilla self-attention, which performs point-wise dot products that effectively assume a zero-lag relationship. Furthermore, real-world data is frequently non-stationary, meaning its statistical properties evolve over time, complicating long-term predictions.

VCformer addresses these limitations by introducing a novel architecture that explicitly models lagged correlations and non-stationary dynamics. By combining Variable Correlation Attention (VCA) with a Koopman Temporal Detector (KTD), VCformer achieves state-of-the-art performance across diverse real-world benchmarks.

Methodology

VCformer adopts an "inverted" Transformer structure where each variable's entire time series is treated as a single token. This allows the attention mechanism to focus specifically on inter-variable dependencies.

Variable Correlation Attention (VCA)

The VCA module replaces standard dot-product attention with a mechanism designed to capture correlations across various time lags. As illustrated in the figure below, variables $v_1$ and $v_2$ may show stronger alignment at specific lags (e.g., $lag=3$) than at $lag=0$.

Conceptualizing variable correlations across different time lags (0, 1, and 3).

To maintain computational efficiency, VCA utilizes the Wiener-Khinchin theorem and Fast Fourier Transforms (FFT). This approach reduces the complexity of calculating cross-correlations from $O(T^2)$ to $O(T \log T)$ per variable pair, making it feasible for high-dimensional datasets.

Koopman Temporal Detector (KTD)

To handle non-stationarity, the authors incorporate the KTD module, inspired by Koopman dynamics theory. This theory posits that complex non-linear systems can be represented as linear systems in a higher-dimensional measurement space.

Architecture of the Koopman Temporal Detector (KTD) for modeling non-stationary temporal dynamics.

The KTD module segments the time series, maps these segments into a Koopman space using an MLP encoder, applies a linear transition matrix to evolve the state, and decodes the result back into the original space. This allows the model to adaptively capture changing temporal patterns.

Experimental Results

VCformer was evaluated on eight major benchmarks, including Electricity Transformer Temperature (ETT), Weather, Traffic, and Exchange rates. The model consistently outperformed recent state-of-the-art models like iTransformer, PatchTST, and TimesNet, achieving the lowest Mean Square Error (MSE) in the majority of scenarios.

Key findings include:

  • Plug-and-Play Versatility: Integrating the VCA module into the older Informer model led to a 36.5% relative reduction in MSE, demonstrating the module's general utility.
  • Specialized Performance: Ablation studies showed that VCA is particularly effective for high-dimensional datasets (e.g., Traffic), while KTD is the primary driver of performance in highly volatile, non-stationary data like currency exchange rates.
  • Efficiency: Despite the $N^2$ complexity relative to the number of variables, FFT-based optimization ensures that VCformer remains competitive in terms of training speed and memory usage.

Discussion and Conclusion

VCformer represents a significant step forward in MTS forecasting by moving beyond the assumption of synchronized variable interactions. By explicitly learning optimal lag lengths through VCA and addressing non-stationarity via KTD, the model provides a robust framework for complex real-world forecasting tasks. Future work may focus on improving the interpretability of learned lags and scaling the architecture to "extreme" multivariate datasets containing tens of thousands of variables.

Summary updated

Abstract

Multivariate time series (MTS) forecasting has been extensively applied across diverse domains, such as weather prediction and energy consumption. However, current studies still rely on the vanilla point-wise self-attention mechanism to capture cross-variable dependencies, which is inadequate in extracting the intricate cross-correlation implied between variables. To fill this gap, we propose Variable Correlation Transformer (VCformer), which utilizes Variable Correlation Attention (VCA) module to mine the correlations among variables. Specifically, based on the stochastic process theory, VCA calculates and integrates the cross-correlation scores corresponding to different lags between queries and keys, thereby enhancing its ability to uncover multivariate relationships. Additionally, inspired by Koopman dynamics theory, we also develop Koopman Temporal Detector (KTD) to better address the non-stationarity in time series. The two key components enable VCformer to extract both multivariate correlations and temporal dependencies. Our extensive experiments on eight real-world datasets demonstrate the effectiveness of VCformer, achieving top-tier performance compared to other state-of-the-art baseline models. Code is available at this repository: https://github.com/CSyyn/VCformer.
BibTeX
@inproceedings{yang2024vcformer,
  author = {Yingnan Yang and Qingling Zhu and Jianyong Chen},
  title = {{VCformer: Variable Correlation Transformer with Inherent Lagged Correlation for Multivariate Time Series Forecasting}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure