A Dataset for Physical and Abstract Plausibility and Sources of Human Disagreement

Annerose Eichel, Sabine Schulte im WaldeACL 2023

TL;DR

This paper introduces a novel dataset for evaluating the plausibility of events, extending beyond physical realism to include abstract concepts. By analyzing human judgments on a large scale, they reveal that abstractness influences perceived plausibility and that human annotators tend to favor plausible events, showing greater disagreement on implausible ones.

What Problem Does This Paper Address?

Natural Language Processing (NLP) models often struggle to understand the nuanced concept of plausibility – whether an event described in text is believable or makes sense in the real world. While previous work has focused on physical plausibility (e.g., "a cat eating a sardine" is plausible, but "rain breaking a belly" is not), this paper highlights a significant gap: the role of abstractness in plausibility. Events involving abstract concepts (e.g., "law prohibiting discrimination") can be plausible, but their plausibility is less understood and harder to model than concrete events. Furthermore, human judgment on plausibility is not always clear-cut, and disagreement among annotators is common, especially for less plausible events. This paper aims to address these limitations by creating a dataset that systematically captures both physical and abstract plausibility and by analyzing the sources of human disagreement.

Methods

The authors constructed their dataset through a multi-stage process:

  1. Extracting Natural Language Triples: They began by extracting Subject-Verb-Object (SVO) triples from a large corpus of English Wikipedia articles. This involved parsing sentences, filtering out irrelevant words (like proper nouns and pronouns), and removing triples containing profanity or offensive connotations.
  2. Assigning Abstractness Ratings: For each word in the extracted triples, they assigned abstractness scores using existing concreteness norms. These scores were then used to categorize each constituent (subject, verb, object) into one of five bins, ranging from "highly abstract" to "highly concrete."
  3. Creating Plausible Triples: They then selected triples that represented combinations of abstract and concrete constituents, aiming for a balanced representation across 27 abstractness combinations (e.g., all concrete, all abstract, mixed). From these, they sampled 1,080 plausible event triples.
  4. Generating Pseudo-Implausible Triples: To create implausible events, they systematically perturbed the plausible triples by replacing one or more constituents with randomly sampled words from the plausible triple collection, ensuring the new triples did not already exist in the plausible set. This resulted in another set of 1,080 pseudo-implausible triples.
  5. Human Annotation: Both the plausible and implausible triples were then presented to human annotators on Amazon Mechanical Turk. Annotators rated each event on a scale of 1 to 5 (from implausible to plausible). Rigorous quality checks and post-processing steps were implemented to filter out unreliable annotations and ensure data quality.

Datasets

The primary dataset introduced in this paper is a new benchmark for physical and abstract plausibility. It consists of:

  • 1,733 event triples (a subset of the initially generated plausible and implausible triples).
  • 15,571 human plausibility ratings collected from crowd-workers.
  • Each triple is annotated with an average of 8.9 ratings.

This dataset is unique in its systematic exploration of abstractness in plausibility and its focus on capturing and analyzing human disagreement.

Key Insights and Ideas

  • Abstractness Matters: The degree of abstractness in event constituents significantly impacts perceived plausibility. More concrete words tend to lead to perceptions of implausibility, possibly because they evoke more fixed mental images that are easily violated.
  • Plausibility Preference: Human annotators generally favor plausible events over implausible ones. This is reflected in the distribution of ratings, which is skewed towards the plausible end of the scale.
  • Disagreement Amplifies Implausibility: Human disagreement is significantly higher for implausible events compared to plausible ones. This suggests that while humans can generally agree on what is plausible, identifying and agreeing on what is implausible is a more challenging and subjective task.
  • Generating Implausibility is Tricky: Automatically generating truly implausible events that are not easily reinterpreted as plausible (even in fictional contexts) is a non-trivial task.

Key Results

  • The dataset contains 1,733 event triples with an average of 8.9 ratings per triple.
  • Annotators favored plausibility, with 68.98% of ratings falling into the plausible categories (4 or 5).
  • The average Inter-Annotator Agreement (IAA) using a soft pairwise Jaccard Coefficient was 0.64, indicating reasonable agreement.
  • Analysis showed that concrete subjects and objects were more likely to be associated with implausible majority votings, while abstract verbs were more common in plausible majority votings.
  • For perturbed implausible triples, agreement was lower, with no clear majority for 80% of abstractness combinations.

Limitations and Future Work

  • Fictional Worlds: The authors acknowledge that their dataset does not explicitly account for triples that might be plausible within specific fictional worlds (e.g., "eat-skyscraper").
  • Annotation Subjectivity: While efforts were made to ensure quality, the subjective nature of plausibility and the potential for errors in crowd-sourced annotations are noted.
  • Language Transfer: Adapting the dataset construction approach to other languages might require modifications to event extraction and abstractness rating availability.

Future work suggested includes:

  • Extending the dataset to include more complex sentence structures and broader contexts to resolve ambiguities and potentially reduce disagreement.
  • Further exploring the relationship between abstractness and plausibility, particularly at the constituent level.
  • Utilizing the dataset for developing and evaluating NLP models that can better understand and predict plausibility, including common-sense reasoning and natural language inference tasks.

Summary updated

BibTeX
@inproceedings{eichel2023dataset,
  author = {Annerose Eichel and Sabine Schulte im Walde},
  title = {{A Dataset for Physical and Abstract Plausibility and Sources of Human Disagreement}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure