Abstract
Data-driven models have demonstrated state-of-the-art performance in inferring the temporal ordering of events in text. However, these models often overlook explicit temporal signals, such as dates and time windows. Rule-based methods can be used to identify the temporal links between these time expressions (timexes), but they fail to capture timexes’ interactions with events and are hard to integrate with the distributed representations of neural net models. In this paper, we introduce a framework to infuse temporal awareness into such models by learning a pre-trained model to embed timexes. We generate synthetic data consisting of pairs of timexes, then train a character LSTM to learn embeddings and classify the timexes’ temporal relation. We evaluate the utility of these embeddings in the context of a strong neural model for event temporal ordering, and show a small increase in performance on the MATRES dataset and more substantial gains on an automatically collected dataset with more frequent event-timex interactions.
Introduction
Inferring the temporal order of events—determining if one action occurred before, after, or simultaneously with another—is a cornerstone of document understanding. While modern NLP models excel at capturing semantic nuances, they frequently struggle with the explicit logic of dates. Standard word embeddings like GloVe or ELMo treat "1992" and "1963" as distinct tokens without an inherent chronological relationship. This paper bridges the gap between rule-based temporal logic and differentiable deep learning by creating neural representations that are explicitly "temporally aware."
Methodology: Learning and Broadcasting Timex Embeddings
The proposed framework follows a two-stage approach to integrate temporal logic into neural architectures:
- Pre-training Timex Embeddings: To teach the model chronological logic, the authors developed a grammar to generate 50,000 synthetic pairs of time expressions, ranging from explicit dates (e.g., "August 2013") to relative phrases (e.g., "two weeks ago"). A character-level biLSTM was trained on these pairs to classify their relationship. Using a character-level model is vital as it allows the system to generalize to unseen years or varying date formats (e.g., "10-12-2014" vs "Oct 12th, '14") that word-level models might treat as out-of-vocabulary.
- Integration and Broadcasting: These pre-trained embeddings are then integrated into a state-of-the-art event ordering model. For every token, standard word and POS embeddings are concatenated with the new time embeddings. Crucially, the model employs a "broadcasting" mechanism: if a time expression modifies an event (e.g., "He peaked in March 1994"), the embedding for the date is broadcast to the associated verb using dependency parse rules, anchoring the event directly to the timeline.
Experimental Results
Chronological Logic
The specialized timex model achieved 97.3% accuracy on the synthetic task, demonstrating that it successfully learned the underlying "math" of time. This significantly outperformed standard GloVe and ELMo baselines in pure chronological ordering.
Benchmark Performance
On the standard MATRES benchmark, the model showed a consistent improvement, moving from a 65.50 to 68.61 F1 score when using ELMo embeddings. However, the impact was more pronounced on a larger, automatically collected "Distant Data" corpus from Gigaword, where event-timex interactions are more frequent.
As shown in the results, when time expressions are "masked" (hidden), performance drops significantly. This suggests that while contextual embeddings like ELMo utilize dates to some extent, they do not understand them as robustly as the specialized embeddings. The most significant gains occur when events are explicitly modified by time expressions, transforming complex semantic inference into a simpler comparison of temporal vectors.
Conclusion
By infusing neural models with specialized temporal embeddings, this framework provides a differentiable way to handle chronological logic. While the impact on specific benchmarks like MATRES is constrained by the density of explicit dates in the data, the substantial gains on the Distant Data corpus highlight the potential of this approach for timeline generation and document-level temporal reasoning. Future work may expand this logic to handle durations and fuzzy temporal sets.
Summary updated