A New Attention Mechanism to Classify Multivariate Time Series

Yifan Hao, Huiping CaoIJCAI 2020 · DOI

TL;DR

This paper introduces a novel "Cross Attention" (CA) mechanism for classifying multivariate time series (MTS) data. By effectively capturing both long-term temporal dependencies and inter-variable relationships, their CA-SFCN model significantly outperforms existing state-of-the-art methods across 14 diverse MTS datasets.

What Problem Does This Paper Address?

Classifying multivariate time series (MTS) data, which involves sequences of multiple variables over time, is crucial in many domains. However, existing methods struggle with two key challenges:

  1. Capturing Long-Range Dependencies: Traditional models often fail to effectively learn relationships between data points that are far apart in time.
  2. Representing Inter-Variable Interactions: The complex interplay and dependencies between different variables within the MTS are often overlooked, leading to a loss of important information.

These limitations hinder the accuracy and robustness of MTS classification systems.

Methods

The authors propose a CA-based Stabilized Fully Convolutional Network (CA-SFCN). The core innovation is the Cross Attention (CA) mechanism, which comprises two modules:

  • Temporal Attention (TA) Module: This module focuses on capturing long- and short-term dependencies within a single time series (variable). It works by transforming the input features into query, key, and value spaces, then calculating attention scores that weigh historical information based on its relevance to the current time step. Crucially, it explicitly considers the temporal order and is designed to be more efficient than recurrent methods.
  • Variable Attention (VA) Module: This module addresses the inter-variable interactions. It takes the output from the TA module and evaluates the dependencies between different variables at each time step, further refining the feature representation.

These two attention modules are integrated into a Stabilized Fully Convolutional Network (SFCN). The SFCN itself uses fully convolutional layers (inspired by FCNs) for feature extraction and incorporates techniques to stabilize training, such as instance-wise batch normalization and a strategy for sampling imbalanced batches.

Datasets

The paper evaluates the proposed CA-SFCN on 14 real-world multivariate time series datasets. These datasets vary in size, number of variables, and temporal length, providing a comprehensive benchmark. The datasets include: Action, Activity, Eeg, Eeg2, Ges, HT Sensor, Ozone, Ara Voice, Daily Sport, Net, Har, Auslan, JVowels, and OHC.

Key Insights and Ideas

  • Attention for MTS is Not One-Size-Fits-All: The paper highlights that existing attention mechanisms (like Global Attention or Recurrent Attention) are not directly applicable or optimal for MTS classification due to the unique challenges of temporal and inter-variable dependencies.
  • Decomposing Attention: The CA mechanism's success stems from its dual focus: explicitly modeling temporal dependencies (TA) and then modeling inter-variable dependencies (VA). This granular approach seems to be key.
  • Efficiency Gains: While attention mechanisms can add computational overhead, the authors demonstrate that their TA calculation, relying on matrix multiplications, is more efficient than traditional RNN sequential processing.
  • Stabilization Matters: The inclusion of stabilization techniques in the SFCN architecture is shown to be important for robust training, especially when dealing with diverse datasets.

Key Results

  • Superior Accuracy: CA-SFCN achieved the highest classification accuracy on 13 out of 14 datasets, outperforming 16 other approaches, including state-of-the-art methods like MLSTM-FCN and AMLSTM-FCN.
  • Significant Improvement: On average, CA-SFCN improved accuracy by approximately 2% compared to the second-best method. The gains were particularly pronounced on datasets where existing methods struggled (accuracy < 0.95).
  • Attention Benefits: SFCN models with attention (GA, RA, CA) consistently outperformed the base SFCN model without attention, underscoring the value of attention for MTS classification. CA-SFCN showed a consistent advantage over GA-SFCN and RA-SFCN.
  • Efficiency: While SFCN without attention is fastest, CA-SFCN demonstrates competitive efficiency, especially compared to other attention-based models, making it a practical choice.

Limitations and Future Work

  • Optimal FCN Configuration: The paper notes that determining the optimal configuration of convolutional layers for SFCN remains an open question.
  • Computational Cost: While more efficient than some alternatives, attention mechanisms do add computational overhead. The authors suggest that CA is an "effective and efficient attention mechanism," implying a good balance.
  • Future Work: The paper's conclusion suggests that CA is a promising direction for MTS classification, implying further exploration and refinement of this mechanism could lead to even better results. The authors also mention that the proposed CA mechanism can advance the performance of different convolutional models, hinting at its generalizability.

Summary updated

BibTeX
@inproceedings{hao2020new,
  author = {Yifan Hao and Huiping Cao},
  title = {{A New Attention Mechanism to Classify Multivariate Time Series}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/277},
}
Figure