CSformer: Combining Channel Independence and Mixing for Robust Multivariate Time Series Forecasting

Haoxin Wang, Yipeng Mo, Kunlan Xiang, Nan Yin, Honghe Dai, Bixiong Li, Songhai Fan, Site Mo

TL;DR

This paper introduces CSformer, a novel Transformer-based model for multivariate time series forecasting that effectively combines channel independence and channel mixing. By employing a two-stage self-attention mechanism with shared parameters and adapters, CSformer achieves state-of-the-art performance on various real-world datasets, addressing the limitations of previous approaches that either oversimplified channel interactions or struggled with information loss.

What Problem Does This Paper Address?

Multivariate time series forecasting (MTSF) is crucial for many applications, but it's challenging due to complex long-term dependencies and interrelations between different time series (channels). Recent deep learning models, particularly Transformers, have shown promise, but there's a debate about how to best handle channel information. Some methods focus on "channel independence," treating each time series separately to reduce noise and irrelevant influences. While this can be effective, it risks losing valuable information about how channels interact. Conversely, "channel mixing" approaches consider these interactions but can be computationally expensive and may introduce noise. This paper addresses the gap by proposing a hybrid strategy that leverages the benefits of both channel independence and channel mixing, aiming to capture rich inter-channel dependencies without sacrificing efficiency or introducing noise.

Methods

CSformer introduces a novel two-stage multi-headed self-attention (MSA) mechanism designed to extract and integrate both channel-specific and sequence-specific information.

  • Dimension-augmented Embedding: Before the attention layers, the input time series data undergoes a dimensionality expansion. This process enhances the sequence dimension without distorting the original data, preparing it for subsequent processing.
  • Two-Stage MSA: The core of CSformer is its two-stage MSA.
  • Channel MSA: This stage applies attention along the channel dimension, similar to iTransformer, but it focuses on inter-channel dependencies at each time step rather than treating the entire series as a single token.
  • Sequence MSA: Following the channel MSA, the features are reshaped, and a second MSA is applied along the sequence dimension. Crucially, the parameters of the Channel MSA and Sequence MSA are shared. This parameter sharing is a key innovation, enabling a more cooperative learning process between channel and sequence information.
  • Adapters: After each MSA stage, adapters (small feed-forward networks) are used. These adapters help optimize the discriminative learning of both channel and sequence features, further enhancing the model's ability to capture nuanced information.
  • Training Strategy: The paper advocates for a training strategy that initially emphasizes channel independence and then incorporates channel mixing, aiming to prevent premature noise introduction while still capturing essential inter-channel relationships.

Datasets

The paper evaluates CSformer on several widely used multivariate time series forecasting datasets:

  • ETT: ETTh1, ETTh2, ETTm1, ETTm2
  • Weather
  • Electricity
  • Solar Energy

These datasets cover diverse domains and are standard benchmarks for MTSF tasks.

Key Insights and Ideas

  • The "Sweet Spot" is Hybrid: The paper strongly suggests that neither pure channel independence nor pure channel mixing is optimal for MTSF. A balanced approach, like the one CSformer proposes, is crucial for robust performance.
  • Parameter Sharing is Key: Sharing parameters between the channel and sequence MSA modules is a significant contribution. This allows the model to learn a more integrated representation, where channel and sequence information mutually inform each other, leading to better performance and reduced parameter redundancy compared to separate modules.
  • Order Matters: The experimental results indicate that applying channel MSA before sequence MSA (C → S) yields better results. This suggests that establishing inter-channel relationships first provides a more robust foundation for subsequent temporal analysis.
  • Adapters Enhance Feature Integration: The inclusion of adapters after each MSA stage is shown to be vital for integrating diverse data characteristics and improving predictive accuracy, especially in datasets with many variables.

Key Results

CSformer achieves state-of-the-art (SOTA) performance across multiple datasets and prediction lengths.

  • Overall Performance: CSformer consistently ranks first in MSE and MAE metrics on the tested datasets, outperforming strong baselines like iTransformer, PatchTST, and Crossformer.
  • Comparison to iTransformer: CSformer shows further gains over iTransformer, suggesting that its approach to integrating channel and sequence information is more effective.
  • Outperforming Channel-Independent Models: CSformer surpasses channel-independent models like PatchTST, highlighting the importance of capturing inter-variable dependencies.
  • Generalization: The model demonstrates good generalization capabilities, performing well even when trained on one dataset and tested on another.

Limitations and Future Work

  • Computational Demand: While CSformer aims for efficiency through parameter sharing, the paper acknowledges the need to investigate more rational ways of combining channel independence and channel mixing to further reduce computational demand.
  • Order of Operations: The paper highlights the importance of the order of MSA application (Channel then Sequence) but suggests further research into optimizing this aspect.
  • Future Research: The authors propose exploring more effective strategies for combining channel independence and mixing, as well as reducing computational complexity. They also aim to investigate the underlying mechanisms of their proposed architecture more deeply.

Summary updated

Abstract

In the domain of multivariate time series analysis, the concept of channel independence has been increasingly adopted, demonstrating excellent performance due to its ability to eliminate noise and the influence of irrelevant variables. However, such a concept often simplifies the complex interactions among channels, potentially leading to information loss. To address this challenge, we propose a strategy of channel independence followed by mixing. Based on this strategy, we introduce CSformer, a novel framework featuring a two-stage multiheaded self-attention mechanism. This mechanism is designed to extract and integrate both channel-specific and sequence-specific information. Distinctively, CSformer employs parameter sharing to enhance the cooperative effects between these two types of information. Moreover, our framework effectively incorporates sequence and channel adapters, significantly improving the model's ability to identify important information across various dimensions. Extensive experiments on several real-world datasets demonstrate that CSformer achieves state-of-the-art results in terms of overall performance.
BibTeX
@inproceedings{wang2025csformer,
  author = {Haoxin Wang and Yipeng Mo and Kunlan Xiang and Nan Yin and Honghe Dai and Bixiong Li and Songhai Fan and Site Mo},
  title = {{CSformer: Combining Channel Independence and Mixing for Robust Multivariate Time Series Forecasting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=9iBMBcP5rk},
}
Figure