Embedding time expressions for deep temporal ordering models

Tanya Goyal, Greg Durrett

Abstract

Data-driven models have demonstrated state-of-the-art performance in inferring the temporal ordering of events in text. However, these models often overlook explicit temporal signals, such as dates and time windows. Rule-based methods can be used to identify the temporal links between these time expressions (timexes), but they fail to capture timexes' interactions with events and are hard to integrate with the distributed representations of neural net models. In this paper, we introduce a framework to infuse temporal awareness into such models by learning a pre-trained model to embed timexes. We generate synthetic data consisting of pairs of timexes, then train a character LSTM to learn embeddings and classify the timexes' temporal relation. We evaluate the utility of these embeddings in the context of a strong neural model for event temporal ordering, and show a small increase in performance on the MATRES dataset and more substantial gains on an automatically collected dataset with more frequent event-timex interactions.

Introduction

While deep learning has mastered many aspects of natural language, it remains surprisingly "time-blind." When a news article says, "He visited France in 1992 and went to Germany in 1963," a human immediately knows the Germany trip happened first. However, standard neural models often treat "1992" and "1963" as just two similar-looking strings or rely on word embeddings that don't capture the numerical and chronological distance between them.

Existing solutions usually fall into two camps: ignoring explicit time signals entirely or using rigid, rule-based post-processing to fix errors. The former misses vital clues, while the latter is hard to integrate into the "end-to-end" differentiable nature of modern neural networks. This research seeks a middle ground: a learned, distributed representation (an embedding) that encodes the actual temporal meaning of expressions like "October 2014" or "three weeks ago."

Methods

The proposed framework operates in two stages:

1. Learning Timex Embeddings

Since real-world datasets with labeled time relations are small, the authors generated synthetic data. They created a grammar of time expressions (e.g., [mm dd, yy], [xx units later]) and sampled 50,000 pairs. A character-level biLSTM was trained to take two strings (like "Jan 93" and "1995") and classify their relationship as before, after, or simultaneous. Using a character-level model is crucial as it allows the model to generalize to unseen dates by picking up on the numerical structure of strings.

Architecture for learning timex embeddings using a character-level biLSTM.

2. Integrating into Event Models

These pre-trained embeddings are then integrated into a state-of-the-art temporal ordering model. For every word in a sentence, its word embedding (GloVe/ELMo) is concatenated with its new timex embedding (or a zero vector if it's not a time expression). To ensure the time information reaches the relevant events, a dependency parser is used to "broadcast" the timex embedding to the modified verb's representation.

Evaluation and Results

The model was evaluated on two primary datasets: MATRES, a standard benchmark for event temporal ordering, and a new Distant Data corpus collected from Gigaword using high-precision rules to target sentences where events are explicitly linked to dates.

Key Findings

  • Intrinsic Accuracy: The character-level timex model achieved 97.3% accuracy on the synthetic task, proving it successfully learned to "read" and compare dates.
  • MATRES Performance: Adding timex embeddings to the baseline model (Cheng and Miyao, 2017) improved mean accuracy from 65.50% to 68.61% when using ELMo.
  • Distant Data Gains: On the dataset with more frequent event-timex interactions, the proposed model outperformed the ELMo baseline by a more substantial margin, demonstrating that the model effectively harvests chronological data when it's available.
Performance comparison on the Distant Data across different training set sizes and embedding types.

The experiments also revealed an "ELMo Gap": even powerful contextual embeddings like ELMo don't organically learn temporal logic well. When time expressions were masked out, ELMo-based models saw a significant performance drop, proving they were relying on the strings themselves rather than a deep understanding of the timeline.

Conclusion

This work demonstrates that specialized timex embeddings can significantly enhance a neural network's sense of time. The primary limitation remains the density of time expressions in natural text; in datasets like MATRES, few event pairs are explicitly anchored to dates. Future work could involve expanding the grammar to handle more complex, relative time expressions (e.g., "the following Tuesday") which require document-level context to resolve. These embeddings could also prove useful for temporal question answering or timeline construction.

Summary updated

Abstract

Data-driven models have demonstrated state-of-the-art performance in inferring the temporal ordering of events in text. However, these models often overlook explicit temporal signals, such as dates and time windows. Rule-based methods can be used to identify the temporal links between these time expressions (timexes), but they fail to capture timexes' interactions with events and are hard to integrate with the distributed representations of neural net models. In this paper, we introduce a framework to infuse temporal awareness into such models by learning a pre-trained model to embed timexes. We generate synthetic data consisting of pairs of timexes, then train a character LSTM to learn embeddings and classify the timexes' temporal relation. We evaluate the utility of these embeddings in the context of a strong neural model for event temporal ordering, and show a small increase in performance on the MATRES dataset and more substantial gains on an automatically collected dataset with more frequent event-timex interactions.
BibTeX
@inproceedings{goyal2019embedding,
  author = {Tanya Goyal and Greg Durrett},
  title = {{Embedding time expressions for deep temporal ordering models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=HhvCnBxI6p7},
}
Figure