Abstract
Complex tasks like question answering need to be able to identify events in text and the relations among those events. We show that this event identification task and a related task, identifying the semantic class of these events, can both be formulated as classification problems in a word-chunking paradigm. We introduce a variety of linguistically motivated features for this task and then train a system that is able to identify events with a precision of 82% and a recall of 71%. We then show a variety of analyses of this model, and their implications for the event identification task.
Introduction
While Natural Language Processing (NLP) has become adept at identifying entities (people, places, organizations), understanding the events those entities participate in remains a significant challenge. For a question-answering system to explain how the 1991 pegging of the Argentine peso led to a 2003 crisis, it must recognize "linking," "contribute," and "crisis" as related events with specific temporal and causal roles. This paper addresses two specific gaps: determining which words describe an event (Event Identification) and assigning a class to those events, such as whether they are a perception, a report, or an intentional action (Semantic Classification).
Methodology
The authors introduce STEP (System for Textual Event Parsing), which formulates event identification as a word-chunking problem, similar to Named Entity Recognition. They utilize the B-I-O (Beginning, Inside, Outside) tagging scheme to label every word in a document. To power this classification, they employ a Support Vector Machine (SVM) fed with a massive array of features, including morphological stems, part-of-speech tags, WordNet hypernyms, and contextual "sliding windows."
The study utilizes the TimeBank corpus, consisting of approximately 200 news documents annotated using the TimeML scheme. This dataset is unique because it marks not just verbs as events, but also nouns ("the investigation") and adjectives ("he was ready") that function as events within a specific context.
Key Insights and Ideas
- Verbs vs. Nouns: The system performs significantly better on verbs (F-measure of 0.88) than on nouns (F-measure of 0.54). Verbs are linguistically "louder" indicators of events, whereas nouns are often more ambiguous.
- The "Said" Phenomenon: The "Reporting" class had the highest performance, largely because the word "said" accounts for 60% of reporting events in the corpus and is an event 98% of the time.
- Feature Utility: Simple textual and part-of-speech features are sufficient for identifying that an event exists. However, identifying the semantic class requires much more complex features, specifically WordNet hypernyms and morphological affixes.
- Diminishing Returns: Learning curves showed that the model plateaus after seeing only about 20% of the training data, suggesting that simply adding more data of the same type will not solve remaining errors.
Key Results
STEP was compared against two baselines: a simple "Memorize" lookup table and "Sim-Evita" (a simulation of a then-state-of-the-art rule-based system). STEP outperformed both baselines in both identification and classification tasks.
STEP achieved 82% precision and 71% recall (F=0.76) for event identification. Semantic class identification proved more difficult, with STEP achieving 67% precision and 51% recall (F=0.58).
Conclusion
The primary limitation of current methods is the data sparsity of nouns, which are less frequently used as events than verbs. While the STEP system demonstrates that event identification can be effectively handled as a chunking task, further progress requires discovering more descriptive features that can capture the subtle distinctions between static states and dynamic events.
Summary updated