Embedding Time Differences in Context-sensitive Neural Networks for Learning Time to Event

Nazanin Dehghani, Hassan Hajipoor, Hadi AmiriACL 2021 · OpenReview

TL;DR

This paper introduces a novel context-sensitive neural network for predicting the time to an event (TTE), particularly for streaming data like tweets, which often lack explicit temporal markers. By incorporating "time difference embeddings" derived from neighboring tweets, their model significantly outperforms existing methods, demonstrating the power of contextual temporal information.

What Problem Does This Paper Address?

Predicting the time until an event occurs (Time to Event or TTE prediction) is crucial for many applications, from generating timelines to summarizing news and even predicting medical events. Traditional methods often rely on explicit temporal expressions within the text. However, this approach falters when dealing with informal, short-form content like tweets, where such expressions are rare. Furthermore, event-related content in streams is often temporally clustered, making it challenging to discern precise timing. This paper tackles the problem of accurately predicting TTE in such noisy, context-dependent streaming data.

Methods

The core of the proposed solution is a context-sensitive neural model that operates within a multi-task learning framework. Instead of just looking at the target tweet, the model leverages temporally-rich neighboring tweets (context tweets) to infer the time to event.

A key innovation is the introduction of time difference embeddings. These are learned using a separate LSTM-based character-level sequence-to-sequence model. This "diff model" takes two timestamps as input and predicts their difference in hours. The resulting embeddings capture the temporal relationship between tweets.

The main TTE model then takes the target tweet, its context tweets, and these learned time difference embeddings as input. It performs two tasks simultaneously:

  1. TTE Sign Classification: Predicting whether the tweet was posted before, at the same time, or after the event.
  2. TTE Value Regression: Estimating the absolute time difference (in hours) to the event.

The authors use a technique to balance the losses from these two tasks, preventing one from dominating the training process.

Datasets

The authors introduce a new multi-genre dataset comprising:

  • English events: Soccer competitions (specifically English Premier League) and Academy Awards ceremonies.
  • Relevant tweets: Collected from Twitter for these events.
  • Dutch tweets: Used for comparison with existing methods.

The EPL dataset was carefully curated, including 42 distinctive hashtags and tweets sent within 14 days of match days. For evaluation, they also used tweets related to the 2018 Academy Awards ceremony and a Dutch dataset.

Key Insights and Ideas

  • Context is King (for temporal prediction): The study strongly suggests that incorporating information from neighboring tweets significantly improves TTE prediction accuracy, especially for tweets lacking explicit temporal markers.
  • Time Differences Matter: Explicitly learning and embedding the time differences between tweets is a powerful way to capture temporal relationships, even when direct temporal expressions are absent.
  • Streaming Data Challenges are Unique: The model's success on Twitter data highlights the need for specialized approaches that can handle the sparsity and temporal clustering typical of such platforms.
  • Early Prediction is Harder but Possible: The research indicates that context tweets are even more helpful for predicting events that are farther in the future, suggesting potential for early event prediction.

Key Results

  • The proposed model is 1.4 hours more accurate on English tweets and 3.3 hours more accurate on Dutch tweets compared to the state-of-the-art.
  • The "diff model" for learning time difference embeddings achieves an impressive 98.3% accuracy in encoding temporal differences.
  • Adding context tweets (k=1, 2, 3) consistently improves performance, with context size 3 yielding the best results (MAE of 4.24 hours on the T-T set).
  • Even tweets without temporal expressions (N set) provide useful lexical clues when used as context, slightly improving performance.
  • Ablation studies show that removing time difference embeddings increases the MAE by 0.8 hours, underscoring their importance.

Limitations and Future Work

  • Data Distribution Differences: The model's performance on the Oscars dataset was lower than on the EPL dataset, attributed to differences in training and test data distributions.
  • Future Work: The authors suggest expanding their model to other temporal tasks, particularly focusing on early prediction of events. They also propose investigating the utility of user or social network information for enhancing TTE prediction.

Summary updated

Abstract

We propose an effective context-sensitive neural model for time to event (TTE) prediction task, which aims to predict the amount of time to/from the occurrence of given events in streaming content. We investigate this problem in the context of a multi-task learning framework, which we enrich with time difference embeddings. In addition, we develop a multi-genre dataset of English events about soccer competitions and academy awards ceremonies, and their relevant tweets obtained from Twitter. Our model is 1.4 and 3.3 hours more accurate than the current state-of-the-art model in estimating TTE on English and Dutch tweets respectively. We examine different aspects of our model to illustrate its source of improvement.
BibTeX
@inproceedings{dehghani2021embedding,
  author = {Nazanin Dehghani and Hassan Hajipoor and Hadi Amiri},
  title = {{Embedding Time Differences in Context-sensitive Neural Networks for Learning Time to Event}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=4UwX3KGpM0},
}
Figure