TL;DR
Crossformer is a novel Transformer architecture designed for multivariate time series (MTS) forecasting that explicitly models the relationships between different variables (cross-dimension dependency). By utilizing a unique two-dimensional embedding and a two-stage attention mechanism, it outperforms existing state-of-the-art models like Informer and Autoformer across numerous real-world benchmarks.
What Problem Does This Paper Address?
In multivariate time series forecasting—predicting future values for multiple correlated variables like temperature, humidity, and wind speed—capturing dependencies is everything. Most existing Transformer-based models focus almost exclusively on cross-time dependency (how a variable relates to its own past).
To do this, they typically "squash" all variables at a single time step into a single vector. This approach has two major flaws:
- Implicit Modeling: It assumes the model will "figure out" the relationships between variables through the embedding, but it doesn't provide an explicit mechanism to do so.
- Information Loss: A single data point in time carries very little information compared to a "segment" or "patch" of time.
Crossformer aims to bridge this gap by treating time and dimensions as two distinct axes that both require explicit attention, shifting the paradigm from point-wise embeddings to segment-wise embeddings.
Methods
The authors introduce three core innovations to the Transformer pipeline to effectively capture both temporal and dimensional dependencies:
- Dimension-Segment-Wise (DSW) Embedding: Instead of embedding all variables at one time step, Crossformer partitions each univariate series into segments (or patches) over time. These segments are then embedded into a 2D vector array. This preserves the "identity" of each dimension while capturing local temporal patterns.
- Two-Stage Attention (TSA) Layer: To process the 2D array efficiently, the model uses two distinct attention stages:
- Cross-Time Stage: Applies attention across time segments within each individual dimension.
- Cross-Dimension Stage: Applies attention across different variables at the same time segment. To keep this scalable for datasets with hundreds of variables, they use a Router Mechanism. A small number of learnable "router" vectors aggregate information from all dimensions and then distribute it back, reducing complexity from $O(D^2)$ to $O(D)$.
- Hierarchical Encoder-Decoder (HED): The model uses a multi-scale approach. In the encoder, adjacent time segments are merged as you go deeper into the network, allowing the model to capture dependencies at increasingly coarse scales (e.g., from hourly to daily patterns). The decoder then aggregates predictions from all these scales to produce the final forecast.
Datasets
The model was evaluated on six widely-used real-world multivariate time series benchmarks:
- ETTh1 & ETTm1: Electricity Transformer Temperature (hourly and 15-minute intervals).
- WTH: Weather data containing 12 meteorological indicators.
- ECL: Electricity Consuming Load for 321 clients.
- ILI: Influenza-Like Illness weekly recorded indicators.
- Traffic: Hourly road occupancy rates from 862 sensors in San Francisco.
Key Insights and Ideas
- Segments over Points: Much like how Vision Transformers (ViT) found that "patches" of images are more informative than individual pixels, Crossformer demonstrates that "segments" of time series are more informative than individual data points.
- The "Router" Efficiency: One of the biggest hurdles in MTS is the "curse of dimensionality." If you have 800 variables, a standard attention mechanism requires 640,000 connections per time step. The Router Mechanism proves you can achieve "all-to-all" communication between variables using a tiny fraction of that computation.
- Scale Matters: By using a hierarchical structure, the model acknowledges that time series patterns exist at multiple resolutions. A sudden spike (fine scale) and a seasonal trend (coarse scale) both matter, and Crossformer's HED architecture captures both.
Key Results
Crossformer demonstrated superior performance across the board:
- Accuracy: It ranked top-1 in 36 out of 58 experimental settings and top-2 in 51 out of 58 settings, consistently beating strong baselines like FEDformer and Autoformer.
- Efficiency: Despite modeling more complex dependencies, Crossformer showed better memory efficiency and faster running times than many competitors, especially as the input sequence length ($T$) increased.
- Ablation Success: The authors proved that both the DSW embedding and the TSA layer were critical; removing either led to a significant drop in forecasting accuracy.
Limitations and Future Work
- Noise in High Dimensions: The current "all-to-all" connection in the Cross-Dimension stage might introduce noise when dealing with hundreds of variables where only a few are actually related. The authors suggest that future work could incorporate "sparse" or "graph-based" attention to filter this noise.
- Permutation Invariance: Like all Transformers, the model is theoretically permutation-invariant. While position embeddings help, concurrent research suggests that for very simple patterns, linear models might still hold an edge because they inherently respect temporal order.
- Dataset Scale: The authors note that while Crossformer is powerful, Transformers generally require larger datasets to truly shine compared to simpler neural networks or statistical models.
Summary updated