Discourse-Level Event Temporal Ordering with Uncertainty-Guided Graph Completion

Jian Liu, Jinan Xu, Yufeng Chen, Yujie ZhangIJCAI 2021 · DOI

TL;DR

This paper introduces UCGraph, a novel approach to ordering events in text by framing it as a graph completion problem. By leveraging graph representation learning and an uncertainty-guided strategy, UCGraph effectively captures inter-event dependencies and improves global consistency in temporal ordering, setting a new state-of-the-art.

What Problem Does This Paper Address?

Understanding the temporal sequence of events in a document is crucial for many natural language processing tasks, such as building timelines or summarizing information. While previous methods have made progress, they often rely on manually crafted features that are time-consuming to create and specific to particular datasets or domains. A key challenge is maintaining global consistency across an entire document, as local predictions can easily lead to cascading errors. Existing approaches struggle to reason over long contexts and learn these complex inter-dependencies without extensive human engineering.

Methods

The core of UCGraph lies in its innovative approach to event temporal ordering:

  • Graph Representation: Each document is treated as a temporal graph where events are nodes and temporal relations (like "before," "after," "simultaneous") are edges. The task then becomes predicting and completing these edges in an initially empty graph.
  • Graph Mask Pre-training: To learn the underlying temporal relationships, the model is trained using a "masked graph" approach. A portion of the temporal graph's edges are randomly masked, and the model learns to reconstruct them by considering the remaining graph structure. This allows the model to assimilate global features and learn inter-dependencies.
  • Uncertainty-Guided Graph Completion: At test time, to mitigate error propagation, UCGraph employs an "uncertainty-first" strategy. It predicts temporal relations for missing edges and then estimates the model's uncertainty for each prediction using Monte Carlo Dropout. The edge with the least uncertainty is selected and added to the graph next. This "certain-first" approach aims to make the most reliable predictions early on, reducing the risk of cascading errors.
  • Relational Graph Convolutional Networks (R-GCNs): The paper utilizes R-GCNs for learning node and graph representations. R-GCNs are adept at modeling relational data and are used to capture the graph-aware representations of events by assimilating information from their neighbors.

Datasets

The paper evaluates UCGraph on the following datasets:

  • TBDense: A dataset for local event temporal ordering.
  • TDDiscourse: The largest discourse-level event temporal ordering benchmark, which includes two subsets:
  • TDD-Man: Augments TimeBank-Dense by manually annotating TLINKs between event pairs that are more than one sentence apart.
  • TDD-Auto: Derives new TLINKs through automatic inference rules.

Key Insights and Ideas

  • Framing as Graph Completion is Powerful: Treating temporal ordering as a graph completion problem allows for the direct application of graph representation learning techniques, which are effective at capturing global dependencies.
  • Uncertainty is Key for Robustness: The "certain-first" strategy, guided by model uncertainty, is crucial for handling the inherent challenges of sequential prediction in graph completion. It helps avoid error propagation by prioritizing the most confident predictions.
  • Pre-training Matters: The graph mask pre-training mechanism significantly improves the model's ability to learn inter-event temporal dependencies without requiring complex feature engineering.
  • Initialized Representations are Flexible: The model's effectiveness is shown to be relatively independent of the initial node representations (whether using BiLSTM or BERT), highlighting the strength of the graph-based approach.

Key Results

  • UCGraph, particularly when combined with BERT for node representations (UCGraph+BERT), achieves state-of-the-art performance on the TDDiscourse benchmark (TDD-Auto and TDD-Man), outperforming previous methods by significant margins (e.g., +2.4% in F1 on average).
  • The approach demonstrates strong performance in maintaining global consistency, a critical aspect of discourse-level event temporal ordering, often outperforming methods that rely on explicit global constraints.
  • The uncertainty-guided strategy is shown to be more effective than random or logit-based ordering strategies for graph completion. Predictions with lower uncertainty values are significantly more likely to be correct.

Limitations and Future Work

  • Local vs. Global: The paper notes that on the TBDense dataset (focused on local ordering), UCGraph underperforms some methods. This suggests that the graph-based approach, designed for global reasoning, might be less advantageous when the graph structure is very small.
  • Future Directions: The authors express interest in applying their method to other discourse-level tasks, such as document-level relation extraction and event causality identification. They also suggest further exploration of different graph completion strategies and uncertainty modeling techniques.

Summary updated

BibTeX
@inproceedings{liu2021discourse,
  author = {Jian Liu and Jinan Xu and Yufeng Chen and Yujie Zhang},
  title = {{Discourse-Level Event Temporal Ordering with Uncertainty-Guided Graph Completion}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/533},
}
Figure