Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting

Yunhao Zhang, Junchi YanICLR 2023 · OpenReview

TL;DR

Crossformer is a novel Transformer architecture designed for multivariate time series (MTS) forecasting that explicitly models the relationships between different variables (cross-dimension dependency). By utilizing a unique two-dimensional embedding and a two-stage attention mechanism, it outperforms existing state-of-the-art models like Informer and Autoformer across numerous real-world benchmarks.

What Problem Does This Paper Address?

In multivariate time series forecasting—predicting future values for multiple correlated variables like temperature, humidity, and wind speed—capturing dependencies is everything. Most existing Transformer-based models focus almost exclusively on cross-time dependency (how a variable relates to its own past).

To do this, they typically "squash" all variables at a single time step into a single vector. This approach has two major flaws:

  1. Implicit Modeling: It assumes the model will "figure out" the relationships between variables through the embedding, but it doesn't provide an explicit mechanism to do so.
  2. Information Loss: A single data point in time carries very little information compared to a "segment" or "patch" of time.
Motivation for segment-wise cross-dimension dependency

Crossformer aims to bridge this gap by treating time and dimensions as two distinct axes that both require explicit attention, shifting the paradigm from point-wise embeddings to segment-wise embeddings.

Methods

The authors introduce three core innovations to the Transformer pipeline to effectively capture both temporal and dimensional dependencies:

  1. Dimension-Segment-Wise (DSW) Embedding: Instead of embedding all variables at one time step, Crossformer partitions each univariate series into segments (or patches) over time. These segments are then embedded into a 2D vector array. This preserves the "identity" of each dimension while capturing local temporal patterns.
  2. Two-Stage Attention (TSA) Layer: To process the 2D array efficiently, the model uses two distinct attention stages:
  3. Cross-Time Stage: Applies attention across time segments within each individual dimension.
  4. Cross-Dimension Stage: Applies attention across different variables at the same time segment. To keep this scalable for datasets with hundreds of variables, they use a Router Mechanism. A small number of learnable "router" vectors aggregate information from all dimensions and then distribute it back, reducing complexity from $O(D^2)$ to $O(D)$.
Two-Stage Attention (TSA) Layer and Router Mechanism
  1. Hierarchical Encoder-Decoder (HED): The model uses a multi-scale approach. In the encoder, adjacent time segments are merged as you go deeper into the network, allowing the model to capture dependencies at increasingly coarse scales (e.g., from hourly to daily patterns). The decoder then aggregates predictions from all these scales to produce the final forecast.

Datasets

The model was evaluated on six widely-used real-world multivariate time series benchmarks:

  • ETTh1 & ETTm1: Electricity Transformer Temperature (hourly and 15-minute intervals).
  • WTH: Weather data containing 12 meteorological indicators.
  • ECL: Electricity Consuming Load for 321 clients.
  • ILI: Influenza-Like Illness weekly recorded indicators.
  • Traffic: Hourly road occupancy rates from 862 sensors in San Francisco.

Key Insights and Ideas

  • Segments over Points: Much like how Vision Transformers (ViT) found that "patches" of images are more informative than individual pixels, Crossformer demonstrates that "segments" of time series are more informative than individual data points.
  • The "Router" Efficiency: One of the biggest hurdles in MTS is the "curse of dimensionality." If you have 800 variables, a standard attention mechanism requires 640,000 connections per time step. The Router Mechanism proves you can achieve "all-to-all" communication between variables using a tiny fraction of that computation.
  • Scale Matters: By using a hierarchical structure, the model acknowledges that time series patterns exist at multiple resolutions. A sudden spike (fine scale) and a seasonal trend (coarse scale) both matter, and Crossformer's HED architecture captures both.

Key Results

Crossformer demonstrated superior performance across the board:

  • Accuracy: It ranked top-1 in 36 out of 58 experimental settings and top-2 in 51 out of 58 settings, consistently beating strong baselines like FEDformer and Autoformer.
  • Efficiency: Despite modeling more complex dependencies, Crossformer showed better memory efficiency and faster running times than many competitors, especially as the input sequence length ($T$) increased.
  • Ablation Success: The authors proved that both the DSW embedding and the TSA layer were critical; removing either led to a significant drop in forecasting accuracy.

Limitations and Future Work

  • Noise in High Dimensions: The current "all-to-all" connection in the Cross-Dimension stage might introduce noise when dealing with hundreds of variables where only a few are actually related. The authors suggest that future work could incorporate "sparse" or "graph-based" attention to filter this noise.
  • Permutation Invariance: Like all Transformers, the model is theoretically permutation-invariant. While position embeddings help, concurrent research suggests that for very simple patterns, linear models might still hold an edge because they inherently respect temporal order.
  • Dataset Scale: The authors note that while Crossformer is powerful, Transformers generally require larger datasets to truly shine compared to simpler neural networks or statistical models.

Summary updated

Abstract

Recently many deep models have been proposed for multivariate time series (MTS) forecasting. In particular, Transformer-based models have shown great potential because they can capture long-term dependency. However, existing Transformer-based models mainly focus on modeling the temporal dependency (cross-time dependency) yet often omit the dependency among different variables (cross-dimension dependency), which is critical for MTS forecasting. To fill the gap, we propose Crossformer, a Transformer-based model utilizing cross-dimension dependency for MTS forecasting. In Crossformer, the input MTS is embedded into a 2D vector array through the Dimension-Segment-Wise (DSW) embedding to preserve time and dimension information. Then the Two-Stage Attention (TSA) layer is proposed to efficiently capture the cross-time and cross-dimension dependency. Utilizing DSW embedding and TSA layer, Crossformer establishes a Hierarchical Encoder-Decoder (HED) to use the information at different scales for the final forecasting. Extensive experimental results on six real-world datasets show the effectiveness of Crossformer against previous state-of-the-arts.
BibTeX
@inproceedings{zhang2023crossformer,
  author = {Yunhao Zhang and Junchi Yan},
  title = {{Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=vSVLM2j9eie},
}
Figure