SemEval-2010 Task 7: Argument Selection and Coercion

James Pustejovsky, Anna Rumshisky, Alex Plotnick, Elisabetta Jezek, Olga Batiukova, Valeria QuochiSemEval @ ACL 2010 · OpenReview

Abstract

James Pustejovsky, Anna Rumshisky, Alex Plotnick, Elisabetta Jezek, Olga Batiukova, Valeria Quochi. Proceedings of the 5th International Workshop on Semantic Evaluation.

2010.

Introduction

In the field of natural language processing (NLP), understanding a sentence often requires more than identifying its surface-level semantic roles. This paper, presented at SemEval-2010, introduces a specialized task designed to help machines decode the "compositional history" of a sentence's meaning. Specifically, the authors formalize the task of Argument Selection and Coercion.

The core problem lies in linguistic nuances that standard semantic role labeling often misses. For example, in the sentence "John began the book," a verb like "begin" typically expects an Event (e.g., reading or writing). However, "book" is an Artifact. To resolve this, the human brain performs "type coercion," shifting the Artifact into an Event context. While humans do this intuitively, computational models struggle with these mismatches. This paper establishes a framework to bridge this gap, moving beyond simple metonymy to address pervasive type-shifting in everyday language.

Methodology

The authors adopted the MATTER methodology, a cyclic development process designed for robust NLP task definition.

The MATTER methodology cycle: Model, Annotate, Train, Test, Evaluate, and Revise.

The MATTER cycle consists of six iterative stages: Modeling the phenomenon, Annotating data, Training algorithms, Testing, Evaluating, and Revising. This ensures that the theoretical model of coercion is consistently refined by empirical results.

Central to this approach is the Brandeis Shallow Ontology (BSO), a flat hierarchy of approximately 30 types (e.g., AGENT, ARTIFACT, EVENT, LOCATION). By using a shallow ontology rather than a deep hierarchy like WordNet, the authors provided enough granularity for machine learning without overwhelming the models with hyper-specific categories. The system identifies coercion by detecting when an argument's natural "Source" type does not match the verb's expected "Target" type (e.g., ARTIFACT → EVENT).

Datasets

Two primary datasets were developed to support this task:

  • English Dataset: Sourced from the British National Corpus (BNC), focusing on six highly "coercive" verbs: arrive at, cancel, deny, finish, hear, and report. It contains roughly 2,000 instances with a 30% coercion rate to prevent models from simply defaulting to a "no coercion" guess.
  • Italian Dataset: Sourced from the PAROLE corpus, featuring 21 verbs and a similar distribution of selection versus coercion.

Unlike traditional datasets that label final meanings, these corpora specifically mark the mismatch between a word's surface type and its contextual use.

Key Results

The task's top-performing submission, the UTDMet system, utilized a Support Vector Machine (SVM) integrated with features from WordNet and PropBank. The results demonstrated that machine learning models can identify these subtle semantic shifts with remarkable precision.

Performance metrics for the UTDMet system across various sub-tasks.

Key performance metrics include:

  • Selection vs. Coercion: The system achieved 95-96% precision and recall in distinguishing whether a type-shift occurred.
  • Joint Type Identification: When identifying the specific types involved in a shift (e.g., a DOCUMENT being treated as an EVENT), the system reached a macro-averaged precision of 86%.
  • Baseline Comparison: The system significantly outperformed a simple baseline (guessing the most common type), which only achieved 22% accuracy on joint type identification.

Discussion and Future Work

The findings suggest that coercion is not a rare linguistic exception but a fundamental, pervasive mechanism for efficient communication. The success of the shallow BSO ontology indicates that targeted, less complex taxonomies can be highly effective for specific semantic tasks.

However, the study has limitations. The English dataset was relatively small, focusing on only six verbs, which may have led to models "over-learning" specific verbal quirks. Future research aims to expand the corpus to include more complex semantic manipulations and test the framework across a wider variety of languages to determine if these coercion patterns are universal. Ultimately, the goal is to move toward identifying these shifts in unrestricted, open-domain text.

Summary updated

BibTeX
@inproceedings{pustejovsky2010semeval,
  author = {James Pustejovsky and Anna Rumshisky and Alex Plotnick and Elisabetta Jezek and Olga Batiukova and Valeria Quochi},
  title = {{SemEval-2010 Task 7: Argument Selection and Coercion}},
  booktitle = {SemEval @ ACL},
  year = {2010},
  url = {https://openreview.net/forum?id=VHqCEQ_StTT},
}
Figure