Introduction
Multivariate time series (MTS) forecasting is critical for applications ranging from urban traffic management to power grid optimization. A fundamental challenge in MTS is that variables rarely exhibit perfectly synchronized behavior; instead, changes in one variable often influence others after a specific time delay, known as a "lag." Traditional Transformer architectures typically rely on vanilla self-attention, which performs point-wise dot products that effectively assume a zero-lag relationship. Furthermore, real-world data is frequently non-stationary, meaning its statistical properties evolve over time, complicating long-term predictions.
VCformer addresses these limitations by introducing a novel architecture that explicitly models lagged correlations and non-stationary dynamics. By combining Variable Correlation Attention (VCA) with a Koopman Temporal Detector (KTD), VCformer achieves state-of-the-art performance across diverse real-world benchmarks.
Methodology
VCformer adopts an "inverted" Transformer structure where each variable's entire time series is treated as a single token. This allows the attention mechanism to focus specifically on inter-variable dependencies.
Variable Correlation Attention (VCA)
The VCA module replaces standard dot-product attention with a mechanism designed to capture correlations across various time lags. As illustrated in the figure below, variables $v_1$ and $v_2$ may show stronger alignment at specific lags (e.g., $lag=3$) than at $lag=0$.
To maintain computational efficiency, VCA utilizes the Wiener-Khinchin theorem and Fast Fourier Transforms (FFT). This approach reduces the complexity of calculating cross-correlations from $O(T^2)$ to $O(T \log T)$ per variable pair, making it feasible for high-dimensional datasets.
Koopman Temporal Detector (KTD)
To handle non-stationarity, the authors incorporate the KTD module, inspired by Koopman dynamics theory. This theory posits that complex non-linear systems can be represented as linear systems in a higher-dimensional measurement space.
The KTD module segments the time series, maps these segments into a Koopman space using an MLP encoder, applies a linear transition matrix to evolve the state, and decodes the result back into the original space. This allows the model to adaptively capture changing temporal patterns.
Experimental Results
VCformer was evaluated on eight major benchmarks, including Electricity Transformer Temperature (ETT), Weather, Traffic, and Exchange rates. The model consistently outperformed recent state-of-the-art models like iTransformer, PatchTST, and TimesNet, achieving the lowest Mean Square Error (MSE) in the majority of scenarios.
Key findings include:
- Plug-and-Play Versatility: Integrating the VCA module into the older Informer model led to a 36.5% relative reduction in MSE, demonstrating the module's general utility.
- Specialized Performance: Ablation studies showed that VCA is particularly effective for high-dimensional datasets (e.g., Traffic), while KTD is the primary driver of performance in highly volatile, non-stationary data like currency exchange rates.
- Efficiency: Despite the $N^2$ complexity relative to the number of variables, FFT-based optimization ensures that VCformer remains competitive in terms of training speed and memory usage.
Discussion and Conclusion
VCformer represents a significant step forward in MTS forecasting by moving beyond the assumption of synchronized variable interactions. By explicitly learning optimal lag lengths through VCA and addressing non-stationarity via KTD, the model provides a robust framework for complex real-world forecasting tasks. Future work may focus on improving the interpretability of learned lags and scaling the architecture to "extreme" multivariate datasets containing tens of thousands of variables.
Summary updated