TL;DR
Current time series foundation models force a binary choice: treat channels as completely independent (CI) or fully dependent (CD). This paper introduces Partial Channel Dependence (PCD) via a lightweight Channel Mask (CM) that blends precomputed channel correlations with two learnable domain parameters. By gating cross-channel attention, this plug-and-play approach significantly boosts performance across forecasting, classification, imputation, and anomaly detection in full, few-shot, and zero-shot settings.
What Problem Does This Paper Address?
Multivariate time series (MTS) datasets exhibit two main types of heterogeneity:
- Explicit heterogeneity: Observable structural differences, such as varying sequence lengths and varying numbers of channels ($C$).
- Implicit heterogeneity: Latent statistical properties, notably the strength of relationships across channels.
While recent time series foundation models (TSFMs) design flexible tokenization schemes to handle explicit shape differences, they typically treat channel dependency as one-size-fits-all—either strictly channel-independent (CI) or uniformly channel-dependent (CD). Forcing a model to share information between unrelated sensors introduces noise, while isolating tightly coupled sensors discards critical mutual dynamics. The authors tackle this gap by enabling models to dynamically adjust the degree of channel dependence per dataset.
Methods
The authors propose Partial Channel Dependence (PCD) implemented through a Channel Mask (CM), denoted as $M \in \mathbb{R}^{C \times C}$. For any Transformer architecture where attention is computed across channels (e.g., iTransformer, UniTS):
$$\text{Attn}(Q, K, V) = \text{Softmax}\left(A \odot \frac{QK^\top}{\sqrt{d_k}}\right)V$$
Where:
- $A = I_{C \times C}$ recovers strict CI,
- $A = \mathbf{1}_{C \times C}$ recovers standard CD,
- $A = M = \sigma(\alpha \cdot \bar{R} + \beta)$ yields PCD.
Key Components of Channel Masking:
- Global Correlation Matrix ($\bar{R}$): Computes pairwise absolute Pearson correlations $|R|$ across the training set (to capture both positive and inverse linear associations), normalized by subtracting the dataset mean: $\bar{R} = |R| - \text{mean}(|R|)$. This acts as an offline, zero-overhead prior representing global inter-channel relationships.
- Domain Parameters ($\alpha, \beta$): Two learnable scalar parameters per dataset that scale ($\alpha$) and shift ($\beta$) the normalized correlation before passing through a sigmoid ($\sigma$). This adapts relative correlation values to absolute dependency weights.
- Channel Dependence Ratio ($r(M)$): A diagnostic metric defined as the average value of the off-diagonal elements in $M$. It provides an interpretable measure (between 0 and 1) indicating whether a dataset prefers CI or CD.
- Masked Channel Prediction: A diagnostic evaluation protocol where a target channel's history is masked, forcing the model to forecast its future using only correlated channels.
Datasets
The authors benchmarked across multiple downstream tasks using standard open datasets:
- Forecasting: 13 benchmarks for single-task setups (ETTh1, ETTh2, ETTm1, ETTm2, Exchange, Weather, ECL, Traffic, Solar-Energy, and PEMS03/04/07/08), plus NN5, ILI, SaugeenRiverFlow, and Hospital in foundation/multi-task setups.
- Classification: 18 datasets from the UEA time series archive (e.g., Heartbeat, JapaneseVowels, PEMS-SF, Blink, FaceDetection).
- Imputation: ECL, ETTh1, ETTm1, and Weather.
- Anomaly Detection: SMD, PSM, MSL, SMAP, and SWaT.
- Synthetic Extreme Benchmark: A synthetic 2-channel sine wave dataset with zero cross-correlation designed to test extreme CI conditions.
No new real-world benchmarks were collected, but the evaluation introduced the synthetic uncorrelated experiment and the Masked Channel Prediction protocol.
Key Insights and Ideas
- Global vs. Local Channel Dependency: The standard self-attention matrix ($QK^\top / \sqrt{d_k}$) estimates local input-dependent channel relations per window, which can be noisy. Element-wise multiplying with $M$ injects a global cross-channel prior derived from the entire training distribution.
- Scalars Beat Matrices for Domain Tuning: The authors experimented with channel embedding vectors and full adjustment matrices to scale $\bar{R}$. Counterintuitively, simple 2-parameter scalar tuning ($\alpha, \beta$) generalized better, avoiding overfitting while requiring minimal compute and memory.
- CD Ratio Predicts Architectural Fit: Datasets with higher learned $r(M)$ (such as PEMS traffic networks) see dramatic gains from CD architectures, whereas datasets with low $r(M)$ (like Weather or ETT) benefit far less, explaining conflicting findings in prior literature regarding whether CI or CD is superior.
- Robustness to Noise and Missingness: Because $\bar{R}$ is computed across long historical horizons, the mask remains stable even when up to 75% of input data points are missing.
Key Results
- Single-Task Forecasting (iTransformer + CM):
- Average MSE reduced by 6.3% across all 13 datasets.
- Massive performance leaps on high-channel datasets: 40.2% MSE reduction on PEMS08 ($C=170$), 19.6% on PEMS07 ($C=883$), and 19.0% on PEMS04 ($C=307$).
- Foundation Multi-Task Model (UniTS + CM):
- Under prompt-tuning (PT), CM yielded improvements across all 20 forecasting tasks (5.4% average MSE reduction) and 13 out of 18 classification tasks (4.3% accuracy increase).
- Outperformed task-specific trained models while retaining a unified parameter set.
- Few-Shot and Zero-Shot:
- Outperformed standard UniTS across 5%, 15%, and 20% few-shot data regimes for forecasting and classification, as well as few-shot imputation and anomaly detection.
- In zero-shot forecasting on unseen datasets (Solar, River, Hospital), choosing domain parameters based on correlation similarity achieved consistent error reductions without fine-tuning.
- Computational Overhead:
- Precomputing $R$ offline adds practically zero inference overhead: inference latency for ECL ($C=321$) was $11.2\text{ ms}$ vs. $11.1\text{ ms}$ for vanilla iTransformer.
Limitations and Future Work
- Architecture Dependency: The Channel Mask is fundamentally designed around the cross-variable attention mechanism ($QK^\top$). It cannot be directly plugged into linear-mapping models (e.g., DLinear) or purely patch-independent architectures without redesigning their core layers.
- Zero-Shot Heuristics for Domain Parameters: When evaluating completely unseen datasets in zero-shot regimes, $\alpha$ and $\beta$ cannot be learned directly. The authors rely on heuristic transfers (e.g., taking parameters from the training dataset with the closest correlation density), which, while effective here, is not an end-to-end learned solution.
- Future Work: Extending the concept of Partial Channel Dependence to non-attention foundation architectures and developing self-supervised mechanisms to infer domain parameters on-the-fly for streaming, unseen time series.
Summary updated