TL;DR
This paper introduces SDE-Attention, a novel framework that enhances SDE-RNNs with channel-level attention mechanisms to improve their performance on irregularly sampled time series with missing data. The proposed attention modules consistently boost accuracy, with the time-varying feature attention (SDE-TVF-L) showing particular robustness across various missingness levels and dataset types.
What Problem Does This Paper Address?
Many real-world applications, such as healthcare monitoring and sensor networks, generate time series data that are not only irregularly sampled (measurements taken at unpredictable times) but also suffer from substantial missing observations. Traditional time series models, including standard RNNs, LSTMs, and GRUs, struggle with such data because they typically assume sequential, complete observations. While some models like GRU-D attempt to handle missingness, they often operate in discrete time and don't fully leverage the continuous nature of latent dynamics. Recent advancements in neural ODEs and SDEs offer a principled way to model continuous-time latent dynamics for irregular data, but their robustness, especially under severe missingness, remains a challenge. Specifically, these models often treat all features uniformly, limiting their ability to adapt to varying importance of different features or time points, and to disentangle contributions when data becomes extremely sparse.
Methods
The core contribution is the SDE-Attention framework, which augments a standard SDE-RNN backbone with three plug-and-play attention modules operating at the latent level:
- Static Channel Attention (SDE-SCHA): This module reweights latent dimensions based on global statistics aggregated across all trajectories in a mini-batch at each time step. It uses a small MLP to generate a gate vector that is applied element-wise to the pre-RNN states. This is a computationally efficient way to adapt feature importance.
- Time-Varying Feature Attention (SDE-TVF): This approach extends channel attention to be fully time-dependent. It uses a temporal encoder (LSTM or Transformer) to produce context-aware hidden states, which are then mapped to a matrix of gates. These gates are applied element-wise to reweight features at each specific time step. This allows for dynamic adaptation of feature importance over time. The paper explores both LSTM (SDE-TVF-L) and Transformer (SDE-TVF-T) variants.
- Pyramidal Attention (SDE-PYR): This module builds multi-scale representations by repeatedly downsampling the sequence, applying self-attention at each scale, and then upsampling and fusing the features. This aims to capture both fine-grained local information and coarse, long-range trends.
The SDE-RNN backbone itself uses a Neural SDE to evolve the latent state between observation times and a GRU cell to assimilate observations. The attention modules are inserted before the GRU update.
Datasets
The models were evaluated on:
- A synthetic Periodic dataset: Generated to specifically test model robustness under varying levels of temporal correlation and missingness.
- UCR (University of California Riverside) datasets: A collection of univariate time series classification datasets.
- UEA (University of East Anglia) datasets: A collection of multivariate time series classification datasets.
The paper notes that these datasets cover various domains, including healthcare, human activity, industrial engineering, and natural sciences.
Key Insights and Ideas
- Latent-space attention is crucial for SDE-RNNs: Even when data is fully observed, attention mechanisms consistently improve or match the performance of the vanilla SDE-RNN, indicating their general benefit.
- Time-varying feature attention (SDE-TVF-L) is a robust default: This module consistently performs well across univariate and multivariate datasets, especially under high missingness, suggesting it's a strong choice for general applicability.
- Static channel attention offers a good trade-off: SDE-SCHA provides a simpler, computationally cheaper alternative that still significantly outperforms the baseline, making it suitable for resource-constrained settings.
- Pyramidal attention's complexity might not always be necessary: While effective on some datasets, SDE-PYR's benefits are less consistent, suggesting that its multi-scale approach might not be universally advantageous, especially for simpler time series.
- Attention helps mitigate performance degradation: As missingness increases, attention-augmented models show a slower decline in performance compared to the baseline SDE-RNN, highlighting their resilience.
Key Results
- UCR Datasets:
- SDE-TVF-L achieved the highest average accuracy, improving performance by approximately 4%, 6%, and 10% over the baseline at 30%, 60%, and 90% missingness, respectively.
- SDE-SCHA also consistently outperformed the baseline and was a close second to SDE-TVF-L.
- Performance degradation with increasing missingness was significantly slower for attention-augmented models.
- UEA Datasets:
- SDE-TVF-L yielded up to a 7% gain in mean accuracy under high missingness.
- Attention models generally outperformed the SDE-RNN baseline, especially at medium to high missingness.
- Periodic Toy Dataset:
- All attention variants reduced interpolation Mean Squared Error (MSE) compared to the SDE-RNN baseline, especially at higher missing rates. SDE-PYR showed the lowest error at 10% observed points.
Limitations and Future Work
- Limited Baselines: The study focuses on SDE-RNNs and doesn't directly compare against strong discrete-time baselines for irregular data (e.g., GRU-D, Transformer models with masking).
- Focus on Missing Rate: The investigation primarily considers shifts in the missing rate, not more complex distribution shifts arising from different populations, sensors, or acquisition protocols.
- Attention Placement: All attention modules are applied to the pre-RNN latent state. The paper suggests exploring attention directly over the latent SDE trajectory or joint channel-temporal attention within the continuous dynamics.
- Future Directions: Integrating SDE-Attention into broader benchmark suites, designing adaptive mechanisms to select or combine attention types per dataset, and exploring its application in high-stakes clinical time series are promising avenues.
Summary updated