TL;DR
This paper introduces DeMa, a novel Mamba-based architecture for multivariate time series analysis that tackles the computational limitations of Transformers. DeMa achieves state-of-the-art performance across various tasks by efficiently disentangling temporal dynamics and cross-variate interactions, even accounting for time delays.
What Problem Does This Paper Address?
Multivariate time series (MTS) analysis is crucial for many intelligent applications, from traffic forecasting to anomaly detection. While Transformers excel at capturing dependencies, their quadratic computational complexity makes them impractical for long-term, large-scale MTS data. Mamba offers a linear-time alternative, but its direct application to MTS is suboptimal due to its inability to explicitly model cross-variate interactions, disentangle temporal and inter-series dynamics, and effectively capture time-lagged dependencies. This paper aims to bridge this gap by proposing a Mamba-based backbone that addresses these limitations while maintaining computational efficiency.
Methods
DeMa introduces a "dual-path" architecture built upon Mamba. The core innovations are:
- Adaptive Fourier Filter (AFF): This component decomposes the MTS into two parts: a "Cross-Time Component" capturing global, low-frequency temporal dynamics within each series, and a "Cross-Variate Component" capturing higher-frequency, window-specific interactions between series. This decomposition helps disentangle entangled patterns.
- DuoMNet Blocks: These blocks form the backbone of DeMa and consist of two parallel paths:
- Temporal Path (Mamba-SSD): This path processes the Cross-Time Component. It uses a Mamba-SSD module designed for series-independent, parallel computation to capture long-range temporal dynamics within each individual variate.
- Variate Path (Mamba-DALA): This path processes the Cross-Variate Component. It employs a novel Mamba-DALA module that integrates "Delay-Aware Linear Attention" to explicitly model cross-variate dependencies, considering both global correlation delays and token-level relative delays.
- Weighted Fusion: The outputs from the temporal and variate paths are fused using a weighted layer to create a comprehensive representation.
- Task-Specific Projection: Lightweight heads are attached for downstream tasks like forecasting, imputation, classification, and anomaly detection.
Datasets
The paper evaluates DeMa on a variety of established datasets across five tasks:
- Long-term Forecasting: Electricity, ETT (ETTh1, ETTh2, ETTm1, ETTm2), Exchange, Traffic, Weather, Solar-Energy.
- Short-term Forecasting: PEMS (PEMS03, PEMS04, PEMS07, PEMS08).
- Imputation: ETTh1, ETTh2, ETTm1, ETTm2, Electricity, Weather.
- Anomaly Detection: Server Machine Dataset (SMD), Mars Science Laboratory rover (MSL), Soil Moisture Active Passive satellite (SMAP), Secure Water Treatment (SWaT), Pooled Server Metrics (PSM).
- Classification: Ten multivariate datasets from the UEA Time Series Classification Archive (e.g., Gesture Recognition, Action Recognition, Audio Recognition, Medical Diagnosis).
Key Insights and Ideas
- Decomposition is Key: The paper strongly suggests that explicitly decomposing MTS into temporal and cross-variate components is crucial for effective modeling, especially when dealing with entangled patterns. The Adaptive Fourier Filter plays a vital role in this.
- Delay Matters in Interactions: Capturing time-lagged dependencies between different time series is not just a minor detail but a critical factor for accurate MTS analysis, particularly in tasks like imputation and anomaly detection where precise alignment is needed.
- Mamba's Potential for MTS: While vanilla Mamba has limitations for MTS, DeMa demonstrates how its core strengths (linear complexity, selective mechanisms) can be adapted and enhanced to create a powerful MTS backbone.
- Synergy of Paths: The temporal and variate paths in DeMa are complementary. The temporal path handles intra-series dynamics, while the variate path captures inter-series relationships, and their combination leads to superior performance.
- Efficiency Gains are Significant: DeMa not only achieves high accuracy but also offers substantial improvements in training time and GPU memory usage compared to Transformer-based models, making it practical for real-world applications.
Key Results
- State-of-the-Art Performance: DeMa achieves state-of-the-art results across all five tested tasks: long-term forecasting, short-term forecasting, imputation, anomaly detection, and classification.
- Forecasting: DeMa consistently ranks in the top two for long-term forecasting, achieving the best results in a significant majority of cases for MSE and MAE. It also shows strong performance in short-term forecasting, outperforming competitors by substantial margins.
- Imputation: DeMa consistently outperforms baselines, with its advantage growing as the missing rate increases, highlighting its ability to leverage fine-grained local and temporal alignment.
- Classification & Anomaly Detection: DeMa achieves the highest average accuracy in classification and the best average F1-score in anomaly detection.
- Efficiency: DeMa offers a favorable accuracy-efficiency trade-off, achieving competitive accuracy with significantly less training time and GPU memory compared to Transformer-based models. For example, it achieves the lowest MSE on a large traffic dataset with moderate training time and manageable memory usage.
- Scalability: DeMa scales linearly with input length, unlike Transformers which exhibit quadratic complexity, making it suitable for long-horizon MTS analysis.
Limitations and Future Work
- Hyperparameter Sensitivity: While DeMa is generally robust, the paper notes that imputation and anomaly detection tasks are more sensitive to the fusion weights (α, β) between the temporal and variate paths. This suggests that task-aware tuning of these weights might be beneficial.
- Future Work: The paper implicitly suggests that further exploration of task-specific fusion strategies and potentially more advanced decomposition techniques could lead to even greater improvements. The success of DeMa also opens avenues for applying this dual-path Mamba architecture to other complex sequential data problems.
Summary updated