Abstract
James Pustejovsky, Anna Rumshisky, Alex Plotnick, Elisabetta Jezek, Olga Batiukova, Valeria Quochi. Proceedings of the 5th International Workshop on Semantic Evaluation.
2010.
Introduction
In the field of natural language processing (NLP), understanding a sentence often requires more than identifying its surface-level semantic roles. This paper, presented at SemEval-2010, introduces a specialized task designed to help machines decode the "compositional history" of a sentence's meaning. Specifically, the authors formalize the task of Argument Selection and Coercion.
The core problem lies in linguistic nuances that standard semantic role labeling often misses. For example, in the sentence "John began the book," a verb like "begin" typically expects an Event (e.g., reading or writing). However, "book" is an Artifact. To resolve this, the human brain performs "type coercion," shifting the Artifact into an Event context. While humans do this intuitively, computational models struggle with these mismatches. This paper establishes a framework to bridge this gap, moving beyond simple metonymy to address pervasive type-shifting in everyday language.
Methodology
The authors adopted the MATTER methodology, a cyclic development process designed for robust NLP task definition.
The MATTER cycle consists of six iterative stages: Modeling the phenomenon, Annotating data, Training algorithms, Testing, Evaluating, and Revising. This ensures that the theoretical model of coercion is consistently refined by empirical results.
Central to this approach is the Brandeis Shallow Ontology (BSO), a flat hierarchy of approximately 30 types (e.g., AGENT, ARTIFACT, EVENT, LOCATION). By using a shallow ontology rather than a deep hierarchy like WordNet, the authors provided enough granularity for machine learning without overwhelming the models with hyper-specific categories. The system identifies coercion by detecting when an argument's natural "Source" type does not match the verb's expected "Target" type (e.g., ARTIFACT → EVENT).
Datasets
Two primary datasets were developed to support this task:
- English Dataset: Sourced from the British National Corpus (BNC), focusing on six highly "coercive" verbs: arrive at, cancel, deny, finish, hear, and report. It contains roughly 2,000 instances with a 30% coercion rate to prevent models from simply defaulting to a "no coercion" guess.
- Italian Dataset: Sourced from the PAROLE corpus, featuring 21 verbs and a similar distribution of selection versus coercion.
Unlike traditional datasets that label final meanings, these corpora specifically mark the mismatch between a word's surface type and its contextual use.
Key Results
The task's top-performing submission, the UTDMet system, utilized a Support Vector Machine (SVM) integrated with features from WordNet and PropBank. The results demonstrated that machine learning models can identify these subtle semantic shifts with remarkable precision.
Key performance metrics include:
- Selection vs. Coercion: The system achieved 95-96% precision and recall in distinguishing whether a type-shift occurred.
- Joint Type Identification: When identifying the specific types involved in a shift (e.g., a
DOCUMENTbeing treated as anEVENT), the system reached a macro-averaged precision of 86%. - Baseline Comparison: The system significantly outperformed a simple baseline (guessing the most common type), which only achieved 22% accuracy on joint type identification.
Discussion and Future Work
The findings suggest that coercion is not a rare linguistic exception but a fundamental, pervasive mechanism for efficient communication. The success of the shallow BSO ontology indicates that targeted, less complex taxonomies can be highly effective for specific semantic tasks.
However, the study has limitations. The English dataset was relatively small, focusing on only six verbs, which may have led to models "over-learning" specific verbal quirks. Future research aims to expand the corpus to include more complex semantic manipulations and test the framework across a wider variety of languages to determine if these coercion patterns are universal. Ultimately, the goal is to move toward identifying these shifts in unrestricted, open-domain text.
Summary updated