Learning Sentence-internal Temporal Relations

Mirella Lapata, Alex Lascarides

Abstract

In this paper, we propose a data-intensive approach for inferring sentence-internal temporal relations, a task critical for NLP applications such as multi-document summarization and question answering. Our method bypasses the need for expensive manual coding by exploiting overt temporal markers (e.g., "after", "while") as proxy labels in large, unannotated corpora. We evaluate our models on a pseudo-disambiguation task and compare their performance against both the TimeBank corpus and human subjects. Our results demonstrate that models trained on noisy, approximate data can achieve performance levels comparable to human agreement, suggesting a viable path for the semi-automatic creation of temporal annotations.

Introduction

Understanding the chronological order of events is a fundamental component of natural language understanding. While humans easily infer that "John kissed the girl he met at a party" implies the meeting preceded the kiss, teaching machines to resolve such implicit temporal links remains a significant challenge.

The primary obstacle in this field is the data bottleneck. Existing human-annotated datasets, such as TimeBank, are often too small to train robust machine learning models. Furthermore, manual temporal annotation is notoriously difficult; human annotators frequently struggle to agree on complex links, leading to low recall. This research addresses these issues by leveraging the abundance of "raw" text, using explicit linguistic markers as a supervision signal to learn the underlying logic of temporal relations.

Methodology

The Proxy Label Approach

The core innovation of this study is the use of temporal markers as self-contained labels. By extracting sentences containing explicit markers (e.g., "The results were announced after the market closed"), we can remove the marker and task a model with predicting it back. This "pseudo-disambiguation" task allows us to utilize massive datasets without manual intervention.

Experimental Tasks

We define two primary tasks for our models:

  1. Interpretation: Given two clauses, the model must identify the missing temporal marker from a set of candidates.
  2. Fusion: Given two clauses and a marker, the model must determine the correct syntactic structure—specifically, which clause is the "main" event and which is the "subordinate" event.

Model Architectures and Features

We compared several probabilistic models to determine the most effective way to represent temporal information:

  • Conjunctive vs. Disjunctive Models: Conjunctive models analyze pairs of features across both clauses. While theoretically superior for capturing interactions, they often suffer from data sparsity. Disjunctive models, which treat clause features as independent, proved more robust in practice.
  • Linguistic Abstraction: To improve generalization, we utilized "Temporal Signatures" (comprising tense, aspect, and voice) and semantic classes for verbs and nouns derived from WordNet and Levin classes. This allows the model to learn that "reporting" an event generally follows the event itself, regardless of the specific verb used.
  • Syntactic Signatures: For the Fusion task, we incorporated features representing the complexity and structure of the parse tree.
  • Ensemble Learning: We employed a "stacking" approach using a C5.0 decision tree to combine predictions from multiple sub-models, allowing the system to weigh different linguistic cues dynamically.

Datasets

Our primary training data was the BLLIP Corpus, a 30-million-word collection of Wall Street Journal text. From this, we extracted approximately 83,810 instances of sentence-internal temporal relations. For benchmarking, we used a subset of 40 pairs evaluated by 198 human volunteers and compared our findings against the gold-standard TimeBank corpus.

Results and Discussion

Task Performance

The ensemble model demonstrated significant improvements over baseline approaches. In the Interpretation Task, the model achieved 70.6% accuracy (69.1% F-score), vastly outperforming the word-based baseline of 48.2%. In the Fusion Task, the model was nearly perfect, achieving 97.3% accuracy, indicating that syntactic structure is a highly reliable signal for clause ordering.

Human-Level Performance

To assess the real-world utility of the model, we compared its performance to human agreement using the Kappa scale (which accounts for chance agreement). The ensemble model achieved a Kappa score of .413, while human subjects achieved .421. This suggests that the model is capturing linguistic nuances and ambiguities at a level nearly identical to that of human readers.

Key Insights

Our experiments revealed that linguistic abstraction—moving from raw words to verb classes—is essential for generalization. Furthermore, we found that simpler disjunctive models often outperform complex conjunctive ones when data for specific word combinations is sparse. Finally, the high accuracy in the Fusion task suggests that machines are particularly adept at identifying the "shape" of subordinate clauses relative to main clauses.

Conclusion and Future Work

This research demonstrates that sentence-internal temporal relations can be learned effectively from unannotated data using markers as proxies. While our current focus is limited to relations within a single sentence, the results provide a strong foundation for future work.

Future iterations will aim to address marker ambiguity (e.g., the temporal vs. causal use of "since") and extend the model to extra-sentential relations by incorporating broader paragraph context. Ultimately, this methodology could be used to "pre-annotate" large-scale corpora, significantly accelerating the development of high-quality temporal datasets for the broader NLP community.

Summary updated

Abstract

In this paper we propose a data intensive approach for inferring sentence-internal temporal relations. Temporal inference is relevant for practical NLP applications which either extract or synthesize temporal information (e.g., summarisation, question answering). Our method bypasses the need for manual coding by exploiting the presence of markers like after", which overtly signal a temporal relation. We first show that models trained on main and subordinate clauses connected with a temporal marker achieve good performance on a pseudo-disambiguation task simulating temporal inference (during testing the temporal marker is treated as unseen and the models must select the right marker from a set of possible candidates). Secondly, we assess whether the proposed approach holds promise for the semi-automatic creation of temporal annotations. Specifically, we use a model trained on noisy and approximate data (i.e., main and subordinate clauses) to predict intra-sentential relations present in TimeBank, a corpus annotated rich temporal information. Our experiments compare and contrast several probabilistic models differing in their feature space, linguistic assumptions and data requirements. We evaluate performance against gold standard corpora and also against human subjects.
BibTeX
@article{lapata2006learning,
  author = {Mirella Lapata and Alex Lascarides},
  title = {{Learning Sentence-internal Temporal Relations}},
  journal = {J. Artif. Intell. Res.},
  year = {2006},
  doi = {10.1613/jair.2015},
}
Figure