TL;DR
AltTS is a new forecasting framework that solves "gradient entanglement" by splitting multivariate time series modeling into two paths: one for stable, individual series trends (Autoregression) and another for complex interactions between series (Cross-Relations). By training these paths using alternating optimization instead of joint training, AltTS prevents noisy cross-variable signals from corrupting stable temporal patterns, achieving state-of-the-art results on long-horizon benchmarks.
What Problem Does This Paper Address?
Multivariate time series forecasting (MTSF) is a balancing act between two very different signals:
- Autoregressive (AR) dynamics: The stable, "within-series" patterns where a variable's past predicts its future (e.g., temperature today depends on temperature yesterday).
- Cross-Relation (CR) interactions: The "between-series" dependencies where variables influence each other (e.g., humidity affecting temperature).
Current state-of-the-art models usually try to learn both simultaneously in a single end-to-end architecture. The authors argue this creates an optimization conflict. CR signals are often intermittent and high-variance; when trained jointly, the noisy gradients from the CR component "bleed into" and corrupt the stable gradients needed to learn AR patterns. This leads to brittle training and poor accuracy, especially as the forecasting horizon grows longer and spurious correlations become more likely.
Methods
The authors propose AltTS (Alternating Time Series), which features two main innovations:
1. Dual-Path Architecture
Instead of one big network, AltTS uses two specialized branches:
- AR Path: Uses a simple linear predictor (specifically RLinear) that operates on each channel independently. This captures the stable, long-term temporal trends.
- CR Path: Uses an inverted Transformer equipped with Cross-Relation Self-Attention (CRSA). Crucially, they apply an "AR Mask" to the attention matrix to ensure this path only learns cross-variable dependencies, preventing it from duplicating the AR path's work.
2. Block Alternating Optimization (AO)
This is the "secret sauce." Rather than updating all parameters at once, AltTS uses a cyclic training schedule:
- Step 1: Freeze the CR path and update the AR parameters for a set number of iterations.
- Step 2: Freeze the AR path and update the CR parameters. In their implementation, they found a ratio of 10 AR updates to 2 CR updates per mini-batch worked best, reflecting the fact that autoregressive patterns are more stable and require more consistent refinement.
Datasets
The framework was evaluated on seven standard Long-term Time Series Forecasting (LTSF) benchmarks:
- Weather: 21 meteorological indicators.
- Traffic: Hourly road occupancy rates from 862 sensors.
- Electricity: Hourly consumption for 321 customers.
- ETT Family: Four datasets (ETTh1, ETTh2, ETTm1, ETTm2) tracking electricity transformer oil temperature and load.
Key Insights and Ideas
- Gradient Entanglement is Real: The authors provide striking visual evidence (Figure 1) showing that in joint training, CR gradients exhibit "exploding variance" compared to AR gradients. Alternating optimization successfully isolates this noise.
- Optimization as a Design Variable: The paper suggests that we have hit a point of diminishing returns with architectural complexity. Instead of building "ever more complex architectures," researchers should treat the training schedule itself as a primary tool for improving model performance.
- Theoretical Grounding: Using the law of total covariance, the authors mathematically prove that alternating optimization reduces the variance of the AR gradient compared to joint training, leading to more stable convergence.
Key Results
AltTS demonstrated dominant performance across the board:
- Top-Tier Performance: Out of 56 evaluation slots (7 datasets × 4 horizons × 2 metrics), AltTS achieved a top-two finish in 49 cases (25 best, 24 second-best).
- Long-Horizon Superiority: The gains were most pronounced at longer horizons (e.g., 720 steps), where traditional models often succumb to accumulated noise.
- Beating Specialized Baselines: AltTS outperformed both pure linear models (like OLinear) and pure Transformer models (like iTransformer), proving that a hybrid approach—when optimized correctly—is superior.
Limitations and Future Work
- Computational Overhead: While the paper doesn't dwell on it, alternating optimization can sometimes increase training time or complexity in implementation compared to standard joint SGD.
- Fixed Schedule: The 10:2 update ratio was determined empirically. Future work could explore adaptive schedules that dynamically adjust the update frequency based on the gradient variance observed during training.
- Architecture Agnosticism: While this paper used RLinear and iTransformer as the base modules, the framework is "architecture-agnostic." Testing AltTS with other backbones (like Mamba or CNNs) remains an open area for exploration.
Summary updated