Identification of Temporal Event Relationships in Biographical Accounts

Lucian Silcox, Emmett TomaiNAACL 2013

TL;DR

This paper explores how well existing methods for identifying temporal relationships between events in text can be applied to biographical accounts, a less structured domain than news articles. While the approach works well on a standard news corpus (TimeBank), it struggles with biographies, suggesting that linguistic differences between these domains significantly impact performance.

What Problem Does This Paper Address?

Extracting temporal information from text is crucial for natural language understanding, especially for tasks like question answering. Narratives often imply relationships between events rather than stating them explicitly. While techniques exist for identifying these temporal relationships, they are typically developed and evaluated on structured domains like news articles (e.g., the TimeBank corpus). This paper investigates whether these established techniques can be successfully applied to biographical accounts, a domain characterized by different writing styles and narrative structures, and identifies potential challenges and areas for improvement.

Methods

The authors aim to reproduce and extend the work of Chambers et al. (2007) by applying their event ordering techniques to a newly created corpus of biographical accounts. They use a Support Vector Machine (SVM) classifier trained on an incrementally built feature set. The feature sets start with basic temporal attributes (tense, aspect, class) and progressively add more complex features, including agreement between event attributes, temporal bigrams, event words, WordNet synsets and lemmas, part-of-speech tags, and features related to sentence structure and prepositional phrases. They evaluate their approach on both the TimeBank corpus and their new Bio Corpus, comparing results at each stage of feature set growth.

Datasets

  • TimeBank Corpus: A standard, fully-annotated corpus of news articles used for evaluating temporal relationship extraction techniques. The authors used version 1.1, containing 186 documents and 3,406 identified event pairs.
  • Bio Corpus: A newly created corpus consisting of 17 biographical accounts from Biography.com, annotated with 1,594 event pairs. This corpus is characterized by a higher density of event relationships compared to TimeBank and a narrative style that often explores one aspect of a subject's life before moving to another, leading to more cross-sentence and cross-paragraph temporal references.

Key Insights and Ideas

  • Domain Matters: Temporal relationship extraction techniques that perform well on news articles (TimeBank) do not directly translate to biographical accounts. The linguistic nuances and narrative structures of biographies present unique challenges.
  • Feature Set Sensitivity: The performance of the SVM classifier is highly sensitive to the feature set used. While simpler feature sets often perform as expected, the addition of more complex features can sometimes lead to a decrease in accuracy, particularly in the biographical domain.
  • Prepositional Phrases are Key (for Biographies): Analysis of individual features suggests that prepositional phrases play a surprisingly significant role in identifying temporal relationships within biographical accounts, while part-of-speech tags seem to hinder performance in this domain.
  • Cross-Corpus Differences: The authors highlight differences in how events are referenced across sentences and paragraphs in biographies compared to news articles, suggesting that these structural variations are a major factor in the performance discrepancies.

Key Results

  • TimeBank Performance: The authors' reproduction of Chambers et al.'s work on the TimeBank corpus achieved an accuracy of 51.97% with Mani's features, outperforming Chambers' reported result by one point. This over-performance continued with Lapata's and Chambers' feature sets, reaching 58.22% with the full set, though this was still lower than Chambers' reported 60.45%.
  • Bio Corpus Performance: On the Bio Corpus, the Mani and Lapata feature sets showed similar or slightly improved results compared to TimeBank. However, the full Chambers feature set resulted in a significant drop in accuracy to 56.65%, indicating a reversal of the expected trend.
  • Feature Analysis on Bio Corpus: Independent analysis of features on the Bio Corpus revealed that part-of-speech features degraded performance, while prepositional phrase features significantly improved accuracy, reaching 57.34% with the full feature set.

Limitations and Future Work

  • Corpus Size: The Bio Corpus is relatively small (17 documents), which may limit the generalizability of the findings.
  • Annotator Agreement: The paper does not discuss inter-annotator agreement, which could be a source of error or bias.
  • Feature Selection: The sheer number of potential feature combinations made a comprehensive feature selection analysis outside the scope of this work.

Future work suggested includes:

  • Expanding the size of the biographical corpus.
  • Validating the corpus through cross-validation with multiple annotators to ensure correctness and mitigate biases.
  • Conducting a more thorough feature selection analysis to identify an optimal subset of features for the biographical domain.

Summary updated

BibTeX
@inproceedings{silcox2013identification,
  author = {Lucian Silcox and Emmett Tomai},
  title = {{Identification of Temporal Event Relationships in Biographical Accounts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure