Classifying Temporal Relations with Rich Linguistic Knowledge

Jennifer D’Souza, Vincent NgNAACL 2013

TL;DR

This paper tackles the challenging task of classifying temporal relations between events and times, moving beyond simplified models to consider all 14 original TimeBank relations. They introduce a novel "knowledge-rich, hybrid" approach that combines sophisticated linguistic features with a rule-based system and a machine learning classifier, achieving a significant reduction in error compared to state-of-the-art methods.

What Problem Does This Paper Address?

Understanding the temporal relationships between events is crucial for many natural language processing tasks, such as narrative understanding, question answering, and summarization. The core problem addressed here is temporal relation classification: given two events or an event and a time expression, determining how they relate in time (e.g., "before," "after," "during," "simultaneous").

Existing approaches often simplify this task by collapsing the 14 distinct temporal relations defined in the TimeBank corpus into just six. This simplification loses valuable nuance. Furthermore, many previous methods rely heavily on basic syntactic and limited semantic features. This paper aims to address these limitations by:

  1. Increasing Granularity: Classifying all 14 TimeBank temporal relations, not a reduced set.
  2. Leveraging Richer Knowledge: Incorporating a wider array of linguistic features, including sophisticated semantic and discourse relations, beyond just morpho-syntactic information.
  3. Combining Strengths: Developing a hybrid approach that merges the interpretability and potential for handling data skew of rule-based systems with the generalization power of machine learning.

Methods

The authors propose a knowledge-rich, hybrid approach that combines a strong baseline learning-based classifier with a manually created rule-based system.

Baseline System: * A 14-class temporal relation classifier trained using SVMmulticlass. * Features: A comprehensive set of 68 linguistic features, including: * Lexical: Word strings, head words, and string identity. * Grammatical: POS tags, prepositions, verb phrases, syntactic dominance, and shortest path in parse trees. * Semantic: Temporal role tokens, WordNet synsets, and VerbOcean relations. * Distance: Whether events are in the same sentence. * DCT related: Relations to the document creation time.

Novel Features (for the hybrid approach): * Pairwise Features: Combining features of event 1 with event 2 (e.g., paired tense, aspect, class values). * Dependency Relations: Features derived from dependency parse trees, capturing syntactic relationships between events. * Webster Relations: Lexical relations (synonyms, related-words, near-antonyms, antonyms) between event head words. * WordNet Relations: Lexical relations (hypernyms, hyponyms, troponyms, similar) from WordNet. * Predicate-Argument Relations: Semantic role relations (directional, manner, temporal, cause) from PropBank. * Discourse Relations: Explicit and implicit discourse relations (e.g., RESTATEMENT, SYNCHRONY) extracted from PDTB-style parsers.

Hybrid Approach: The core innovation lies in how rules and machine learning are combined. They explore three methods:

  1. Using rules' predictions as additional features for the classifier.
  2. Applying high-accuracy rules first; if no rule applies, then use the learned classifier.
  3. Similar to method 2, but rules are not used as features during classifier training.

The manual rules are developed through a data-driven process, inspecting TimeBank annotations and refining rules based on observations.

Datasets

  • TimeBank (v1.2) corpus: This is the primary dataset used for both training and evaluation. It consists of 183 newswire articles with annotated events, times, and temporal relations. The paper specifically uses all 14 relation types defined in TimeBank.

Key Insights and Ideas

  • Granularity Matters: Moving from 6 to 14 temporal relations significantly increases the task's complexity but also captures more nuanced temporal information.
  • Rich Linguistic Knowledge is Key: Sophisticated features derived from semantics (predicate-argument, WordNet) and discourse (PDTB) are crucial for improving performance, especially for challenging relations.
  • Hybrid Approach Outperforms Pure Learning: Combining rule-based reasoning with machine learning is more effective than relying solely on machine learning. This suggests that human-derived insights encoded in rules can complement statistical models, particularly for handling data skew and minority classes.
  • Rule Application First is Effective: Applying high-accuracy rules before resorting to the machine learning classifier yields the best results, indicating that rules can effectively handle a subset of cases, allowing the classifier to focus on more ambiguous instances.
  • Discourse Relations are Powerful: Explicit and implicit discourse relations significantly improve classification accuracy, highlighting their importance in understanding temporal flow across sentences.
  • Counter-intuitive Findings: WordNet and predicate-argument relations, surprisingly, did not improve performance. The authors speculate this might be due to the imperfect quality of the analyzers used to extract these features.

Key Results

The paper demonstrates the effectiveness of their hybrid approach, achieving:

  • A 15-16% relative reduction in error compared to the best-performing learning-based baseline system.
  • The best performance was achieved by the "Rules + Features + Rules as Features" architecture (method 3 described above), yielding an accuracy of 53.4% and a macro F-score of 36.6%.
  • The hand-crafted rules themselves achieved a precision of 78.3% and a recall of 15.3% on the instances they were applicable to.
  • Discourse relations provided a significant improvement in both accuracy and macro F-score (p < 0.05).
  • The system showed improvements for both event-event (16-25% relative error reduction) and event-time (6-9% relative error reduction) classifications.

Limitations and Future Work

  • Feature Extraction Quality: The authors acknowledge that the lack of improvement from WordNet and predicate-argument relations might be due to the imperfect quality of the analyzers used to extract these features. Future work could involve using gold-standard quality features to re-evaluate their utility.
  • Rule Coverage: While rules are effective, their recall is relatively low (15.3%), meaning they only handle a fraction of the instances. Future work could focus on increasing rule coverage.
  • Rule Development Effort: The manual rule creation process is labor-intensive. Exploring more automated or semi-automated methods for rule discovery could be beneficial.
  • Generalizability: While tested on TimeBank, further evaluation on other corpora would be valuable to assess the generalizability of the approach.

The authors also emphasize that hybrid rule-based and learning-based approaches are promising for complex reasoning tasks and hope they will receive more attention. They are releasing their set of hand-crafted rules to encourage further research.

Summary updated

BibTeX
@inproceedings{dsouza2013classifying,
  author = {Jennifer D’Souza and Vincent Ng},
  title = {{Classifying Temporal Relations with Rich Linguistic Knowledge}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure