Evita: A Robust Event Recognizer For QA Systems

Roser Saurí, Robert Knippen, Marc Verhagen, James PustejovskyEMNLP 2005

TL;DR

This paper introduces Evita, a novel event recognition system designed for Question Answering (QA) systems. Unlike previous approaches, Evita is domain-independent and doesn't rely on pre-defined event lists, instead using a combination of linguistic and statistical methods to identify and tag event expressions and their associated grammatical features. This flexibility makes it a valuable tool for understanding temporal and intensional relationships in text, crucial for advanced QA.

What Problem Does This Paper Address?

Information Extraction (IE) tasks, particularly event recognition, are fundamental for enabling sophisticated Question Answering (QA) systems. While existing methods have achieved success in specific domains by using pre-defined ontologies of event types, this approach is inherently domain-dependent and lacks flexibility. Furthermore, many QA systems struggle with understanding nuanced linguistic contexts, such as intensional contexts (e.g., attempts, intentions) and temporal relations, which are critical for answering non-factoid questions. These contexts can alter the truthfulness or factuality of propositions, leading to incorrect answers if not properly handled. This paper addresses the need for a more robust, domain-independent event recognition system that can capture these crucial temporal and intensional aspects of events in text, thereby improving the ability of QA systems to understand and respond to complex queries.

Methods

Evita employs a multi-stage approach combining linguistic and statistical techniques:

  • Preprocessing: The system starts with a standard preprocessing pipeline that includes part-of-speech (POS) tagging, lemmatization, and chunking using the Alembic Workbench. This stage extracts essential grammatical information like POS tags, phrase chunks, lexical stems, and syntactic structures.
  • Event Identification: Evita identifies event candidates from verbs, nouns, and adjectives.
  • Verbal Events: Identified through lexical look-up and minimal contextual parsing to filter out weak stative predicates and generics.
  • Nominal Events: Identified through a two-phase process: lexical lookup using WordNet to find nominal event synsets, followed by a disambiguation step using a Bayesian classifier trained on SemCor for ambiguous cases.
  • Adjectival Events: Identified conservatively by tagging adjectives that appear as predicative complements of copular verbs, based on annotations in TimeBank1.2.
  • Grammatical Feature Identification: Evita analyzes several grammatical features associated with identified events:
  • Polarity: Determined by basic pattern matching.
  • Modality: Extracted using pattern matching on modal auxiliaries and adverbials.
  • Tense and Aspect: Derived directly from the morphology of verbal constituents.
  • Non-finite Morphology: Also derived from verbal morphology.
  • Event Class: This is the most challenging feature, currently addressed by a preliminary approach of tagging events with the most frequent class from TimeBank1.2.
  • Chunk Clustering: A crucial step where Evita clusters together independent constituents that refer to the same event. This is particularly important for complex verbal phrases involving modal auxiliaries or split auxiliary forms, and for linking copular verbs with their predicative complements. This clustering is activated by specific triggers and uses finite-state automata compiled from regular expressions.

Datasets

  • TimeBank1.2: This is the gold standard corpus used for evaluation. It consists of approximately 200 news reports annotated with TimeML temporal and event information.
  • SemCor: Used for training the Bayesian classifier for disambiguating nominal events.
  • WordNet: Used for lexical lookup to identify nominal event candidates.

Key Insights and Ideas

  • Domain Independence is Achievable: Evita demonstrates that robust event recognition can be achieved without being tied to a specific domain or a pre-defined ontology of event types. This significantly broadens its applicability.
  • Linguistic and Statistical Synergy: The paper highlights the effectiveness of combining explicit linguistic knowledge (for local contexts and morphology) with statistical methods (for disambiguation) to tackle complex NLP tasks like event recognition.
  • Grammatical Features are Key: Identifying not just the event itself but also its associated grammatical features (tense, aspect, modality, polarity) is crucial for understanding temporal and intensional relationships, which are vital for advanced QA.
  • Chunk Clustering for Complex Structures: The need for chunk clustering to correctly identify events in complex syntactic structures (e.g., modal verbs, split auxiliaries) is a significant practical challenge that Evita addresses.
  • Event Class is the Hardest: The authors acknowledge that classifying event types is the most challenging aspect, and their current approach is preliminary, suggesting that advanced word sense disambiguation techniques are needed.

Key Results

  • Overall Performance: Evita achieves an F-measure of 80.12% (with β=0.5) on event recognition, comparable to inter-annotation agreement scores among human annotators with basic training.
  • Grammatical Feature Accuracy: The accuracy for identifying grammatical features is generally high:
  • Polarity: 98.26%
    • Aspect: 97.87%
    • Modality: 97.02%
    • Tense: 92.05%
    • Non-finite Morphology: 89.95%
    • Class: 86.26%
  • Limitations in Feature Accuracy: The slightly lower accuracy for tense and non-finite morphology is attributed to potential errors in the POS tagger used during preprocessing. The accuracy for event class is the lowest, as it's the only feature not directly derivable from surface cues.

Limitations and Future Work

  • Event Class Classification: The current method for event class identification is basic and relies on frequent annotations from TimeBank1.2. Future work will focus on improving this by experimenting with advanced learning techniques and word sense disambiguation.
  • Non-Local Polarity: Evita's polarity detection is limited to elements within the event chunk and does not account for non-local forms of expressing polarity in English.
  • Event Participants: Evita does not identify event participants but is designed to interface with entity taggers for argument linking.
  • Future Directions: The authors envision enhancing event classification and improving the disambiguation of nominal events. They also plan to explore how Evita can be integrated into larger systems for narrative understanding, summarization, and building factual databases.

Summary updated

BibTeX
@inproceedings{sauri2005evita,
  author = {Roser Saurí and Robert Knippen and Marc Verhagen and James Pustejovsky},
  title = {{Evita: A Robust Event Recognizer For QA Systems}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
}
Figure