Abstract
This paper introduces a novel continuous-time attention mechanism that integrates seamlessly with Neural Differential Equations (NDEs). By replacing discrete summations with continuous integrals, the proposed Att-NDE framework can attend to hidden states at any point in time, making it exceptionally robust to irregularly sampled data and long-range dependencies.
Introduction: The Challenge of Irregular Data
In the idealized world of machine learning tutorials, data arrives in neat, evenly spaced intervals. In the real world—think ICU heart rate monitors, erratic weather sensors, or the flow of a conversation—data is "irregular." It is full of gaps, varying sampling rates, and missing values.
Standard sequential models like Recurrent Neural Networks (RNNs) and Transformers struggle here because they are fundamentally discrete; they expect a "next" step. While Neural Differential Equations (NDEs) were recently introduced to model these sequences as continuous flows, they lacked a way to "focus" on specific past events. In their AAAI-21 paper, Jen-Tzung Chien and Yi-Hsiang Chen introduce Att-NDE to bridge this gap.
Methodology: Continuous-Time Attention
The core innovation is the transition from discrete attention to a continuous-time domain. In traditional attention, a context vector is calculated by summing weighted hidden states. Att-NDE replaces this sum with an integral over time. This allows the model to calculate a context vector for any desired time point, effectively interpolating what the attention should be even where data is missing.
Simultaneous ODE Solving
To make this approach computationally feasible, the authors model the dynamics of three components simultaneously using an ODE solver:
- The hidden state $\mathbf{z}(t)$ (the memory of the sequence).
- The context vector $C(t)$ (the accumulated attentive information).
- The attention score $A(t)$ (the cumulative weight).
By treating the sequence as a continuous function rather than a static matrix, the model can query the past at any arbitrary time resolution. This integration is achieved via the Leibniz integral rule, embedding the physics of time directly into the model's architecture.
Experimental Evaluation
The authors evaluated Att-NDE across three distinct domains to prove its versatility:
- Human Activity Dataset: 3D positions of sensors on subjects performing 11 different actions. This is a classic irregular time-series task.
- MELD (Multimodal EmotionLines Dataset): Dialogue snippets from the TV show Friends, where utterances are treated as irregular events.
- PhysioNet: A healthcare dataset containing 48 hours of ICU physiological signals used to predict mortality.
Key Results and Insights
- Superior Accuracy: On the Human Activity task, Att-NDE achieved 86.8% accuracy, outperforming the previous state-of-the-art "Latent ODE" (84.6%) and standard NDEs.
- Resilience to Missing Data: One of the most striking findings is that Att-NDE can "fill in" missing regions. If a chunk of data is deleted, Att-NDE’s continuous dynamics allow it to maintain a high attention weight in that "empty" region, whereas discrete models lose that information.
- Healthcare and Dialogue Performance: On the PhysioNet mortality prediction task, Att-NDE reached an AUC of 0.833, significantly higher than the RNN baseline (0.783). In sentiment classification (MELD), the continuous self-attention version achieved the highest F1-score (0.565).
Conclusion and Future Work
While Att-NDE provides a robust framework for continuous-time modeling, it faces computational efficiency hurdles. In the self-attention setting, the model currently solves an ODE for every single query vector individually, which is memory-intensive. Future research aims to optimize the memory footprint, extend the framework to Neural Stochastic Differential Equations (SDEs) to better handle noise, and apply the model to finer-grained linguistic data.
Summary updated