Using Query Patterns to Learn the Duration of Events

Andrey Gusev, Nathanael Chambers, Divye Raj Khilnani, Pranav Khaitan, Steven Bethard, Dan JurafskyIWCS 2011 · OpenReview

Andrey Gusev, Nathanael Chambers, Divye Raj Khilnani, Pranav Khaitan, Steven Bethard, Dan Jurafsky Proceedings of the Ninth International Conference on Computational Semantics (IWCS 2011)

Abstract

Andrey Gusev, Nathanael Chambers, Divye Raj Khilnani, Pranav Khaitan, Steven Bethard, Dan Jurafsky. Proceedings of the Ninth International Conference on Computational Semantics (IWCS 2011).

2011.

Introduction

In natural language processing (NLP), "world knowledge" remains a significant challenge. While modern systems can identify parts of speech or basic semantic roles, they often lack the common sense to understand temporal constraints—for instance, knowing that a "meeting" typically lasts an hour, whereas a "reign" spans decades. This temporal information, known as aspectual knowledge, is vital for constructing accurate event timelines and resolving coreference issues, such as determining if two mentions of an event refer to the same occurrence.

The primary obstacle in this field is the scarcity of human-annotated data. The standard TimeBank corpus, for example, contains only 58 documents. Furthermore, event durations are not static values but distributions; a "strike" might last days or months, but rarely seconds. This research introduces an unsupervised method to acquire these duration distributions at scale by leveraging the vast linguistic data available on the web.

Methodology: The Web-Query Approach

The researchers compared a supervised baseline—a Maximum Entropy classifier using linguistic features like tense, aspect, and WordNet hypernyms—against a novel unsupervised web-query system. Using Yahoo! search, the system identifies the frequency of events within specific "duration-revealing" patterns:

  • Coarse-grained patterns: Queries like "<event> yesterday" help identify events likely lasting less than 24 hours.
  • Fine-grained patterns: Templates such as "<event> for * <bucket>" (e.g., "ran for * hours") or "spent * <bucket> <eventing>" (e.g., "spent * minutes washing") provide more granular data.

Normalization and the Iterative Bias

A critical challenge in web-counting is "iterative" or habitual language. For example, the phrase "he has been saying that for years" is common, even though the physical act of "saying" takes only seconds. To correct for this, the authors implemented a normalization step. They divided event-specific counts by the global frequency of that time bucket, effectively measuring if an event is unusually frequent within a specific duration.

Duration distributions for the verbs "to smile" and "to run," showing how different events peak at different temporal buckets.

Key Insights: Distributions Over Means

The study posits that modeling events as single average durations is insufficient. By capturing a full distribution (e.g., 10% seconds, 70% minutes, 20% hours), the system reflects the inherent variability of real-world activities.

Duration distributions for the verbs "to smile" and "to run," showing how different events peak at different temporal buckets.

Another key finding involved the "Context Gap." Through a Mechanical Turk study, the authors found that human accuracy in estimating duration jumps by 20% when provided with the event's subject and object (e.g., "saw the goal" vs. "saw Europe"). However, adding this context to web queries often leads to "data sparsity," where the search engine returns zero results for highly specific phrases.

Results and Evaluation

The unsupervised web-count system demonstrated remarkable performance, particularly in fine-grained tasks:

  • Fine-Grained Accuracy: The unsupervised system achieved 66.5% accuracy in choosing between specific time buckets (seconds, minutes, hours, etc.), outperforming the best supervised model, which reached 62.4%.
  • Coarse-Grained Accuracy: On simpler tasks (determining if an event lasts more or less than a day), the web-pattern approach achieved over 70% accuracy.
  • Human Comparison: Humans given only an isolated verb performed poorly (~42% accuracy), highlighting the importance of the world knowledge captured by the web-query method.

Conclusion and Future Work

While the unsupervised approach successfully bypasses the need for manual annotation, "contextual sparsity" remains a hurdle. The web, despite its size, cannot provide statistically significant counts for every specific subject-verb-object combination. Future research may involve using semantic clusters or thesauri to group similar entities (e.g., grouping all "professional [people]" who "[create] [documents]"), potentially bridging the gap between general world knowledge and specific document understanding. As a result of this study, the authors have released an Event Duration Lexicon containing predicted distributions for 10,000 events.

Summary updated

BibTeX
@inproceedings{gusev2011using,
  author = {Andrey Gusev and Nathanael Chambers and Divye Raj Khilnani and Pranav Khaitan and Steven Bethard and Dan Jurafsky},
  title = {{Using Query Patterns to Learn the Duration of Events}},
  booktitle = {IWCS},
  year = {2011},
  url = {https://openreview.net/forum?id=maj9ympqYY},
}
Figure