The Possible, the Plausible, and the Desirable: Event-Based Modality Detection for Language Processing

Valentina Pyatkin, Shoval Sadde, Aynat Rubinstein, Paul Portner, Reut Tsarfaty

TL;DR

This paper introduces a novel "event-based modality detection" task for Natural Language Processing (NLP), aiming to identify not just modal words but also the specific events they modify and their fine-grained modal senses. By proposing a unified taxonomy and a new annotation standard, the authors pave the way for more nuanced understanding of modality in text, crucial for downstream NLP tasks like uncertainty detection and factuality analysis.

What Problem Does This Paper Address?

Modality in language refers to how we express concepts like possibility, necessity, desire, or obligation. While crucial for understanding nuances in text (e.g., distinguishing between "The event happened" and "The event might have happened"), current NLP approaches to modality detection face several challenges. Existing methods often rely on a limited set of "modal triggers" (like auxiliary verbs), use inconsistent and often linguistically opaque sense labels, and fail to explicitly link modal expressions to the specific events they modify. This fragmentation makes it difficult to leverage modality information effectively in downstream NLP applications. The paper aims to bridge this gap by proposing a more comprehensive and linguistically grounded approach.

Methods

The core contribution is the definition of an event-based modality detection task. This task involves three key components:

  1. Identifying modal triggers: These are the words or phrases that express modality, and crucially, the paper proposes that these can be of any syntactic class, moving beyond traditional focus on auxiliary verbs.
  2. Classifying modal senses: The authors introduce a hierarchical taxonomy of modal senses, harmonizing existing linguistic theories and previous NLP efforts. This taxonomy distinguishes between Plausibility (based on knowledge, world state, or agent ability) and Priority (based on norms, desires, or plans).
  3. Associating triggers with events: The modal trigger and its sense are linked to the specific event they modify, treating modal expressions as semantic modifiers of eventive heads, inspired by Neo-Davidsonian semantics.

For implementation, the authors fine-tune a RoBERTa-base model for sequence labeling tasks, exploring different configurations for trigger detection, sense classification, and joint event-modal detection.

Datasets

The primary dataset used is the Georgetown Gradable Modal Expressions (GME) Corpus. This corpus is derived from the MPQA Opinion Corpus and has been annotated with detailed information about modal expressions, including their senses, the propositions they apply to, and their context. The authors also mention replicating experiments on the dataset used by Marasović and Frank (2016) for comparison.

Key Insights and Ideas

  • Modality is pervasive and diverse: The GME corpus reveals that modal triggers are present in a vast majority of documents (96%) and sentences (48%), highlighting the importance of this linguistic feature.
  • Beyond auxiliary verbs: The paper strongly advocates for treating modal triggers as open-ended, not restricted to a closed class of words, which significantly expands the scope of modality detection.
  • Hierarchical taxonomy is key: The proposed taxonomy provides a more nuanced and linguistically grounded way to categorize modal senses, moving beyond simplistic "epistemic/deontic/dynamic" distinctions.
  • Event-centric approach is crucial: Linking modal expressions directly to the events they modify is essential for practical applications, as sentences can contain multiple events with different modal perspectives.
  • Context is king for sense disambiguation: While trigger lemmas are somewhat informative, the surrounding context (especially the event head) significantly improves the accuracy of modal sense classification.

Key Results

  • RoBERTa outperforms baselines: Fine-tuning a RoBERTa-based model significantly improves performance across all tasks (modal trigger detection, sense classification, and event detection) compared to simpler baselines like majority vote.
  • Task complexity: Detecting modal triggers is relatively easier than classifying their fine-grained senses. Jointly detecting modal triggers and their associated events shows promising results, with only a slight performance drop for fine-grained senses.
  • Performance on fine-grained senses: While challenging, the models achieve respectable F1 scores for fine-grained sense classification, with some senses (like "agent") performing surprisingly well despite being less frequent.
  • Error analysis suggests annotation limitations: Manual inspection of errors indicates that the gold annotations, while precise, might have incomplete coverage, suggesting that the model's actual precision could be higher.

Limitations and Future Work

  • Data Sparsity: The authors acknowledge data sparsity for certain fine-grained senses (e.g., "Desires & Wishes" and "Plans & Goals" were conflated into "Intentions" for some experiments).
  • Annotation Quality: The error analysis suggests that the current gold annotations might not be exhaustive, implying that further annotation efforts could improve model performance.
  • Future Work: The paper proposes this event-based modality detection task and standard as a potential extension for existing semantic representations (like AMR, SDG, UCCA) to better incorporate modal events into downstream NLP tasks. They also suggest exploring more sophisticated models and further refining the annotation scheme.

Summary updated

BibTeX
@inproceedings{pyatkin2021possible,
  author = {Valentina Pyatkin and Shoval Sadde and Aynat Rubinstein and Paul Portner and Reut Tsarfaty},
  title = {{The Possible, the Plausible, and the Desirable: Event-Based Modality Detection for Language Processing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=LCMNR6Wg9Xu},
}
Figure