SemEval-2022 Task 3: PreTENS-Evaluating Neural Networks on Presuppositional Semantic Knowledge

Roberto Zamparelli, Shammur Chowdhury, Dominique Brunato, Cristiano Chesi, Felice Dell’Orletta, Md. Arid Hasan, Giulia VenturiNAACL 2022

TL;DR

This paper introduces PreTENS, a new benchmark for evaluating how well neural networks understand presuppositional semantic knowledge, specifically the relationship between two noun phrases in a sentence. The task challenges models to identify when these relationships are violated, a crucial aspect of true language understanding that goes beyond simple syntax. The results show that while current transformer models are powerful, there's still significant room for improvement in this nuanced area of semantics.

What Problem Does This Paper Address?

Computational linguistics has made great strides in understanding syntax and even some semantic phenomena. However, a deeper level of understanding, often referred to as "metalinguistic awareness," remains elusive for artificial systems. This includes the ability to recognize subtle linguistic ill-formedness that arises not from grammatical errors, but from violations of implicit assumptions or "presuppositions."

PreTENS focuses on a specific type of presupposition: the requirement for two nominal arguments in a sentence to be in a particular taxonomic relationship (e.g., one being a subset of the other, like "dogs" and "animals"). Many common linguistic constructions, such as comparatives ("more than"), coordinations ("except"), and verbs like "prefer," rely on these presuppositions. When these presuppositions are flouted, the sentence can sound "wrong" or "deviant" to a human, even if it's grammatically correct. The challenge is to build models that can detect these semantic violations, which requires not just linguistic knowledge but also world knowledge and common sense.

Methods

The PreTENS task was designed with two sub-tasks:

  • Binary Classification (Sub-task 1): Models had to predict a simple "acceptable" or "unacceptable" label for each sentence.
  • Regression (Sub-task 2): Models predicted a Likert scale score (1-7) representing the degree of acceptability, providing a more nuanced evaluation.

The sentences for the task were artificially generated in English, Italian, and French. The core of the task involved constructions that either enforce or violate specific taxonomic relationships between two argument nouns (A1 and A2). For example, "I like dogs, except cats" is unacceptable because "cats" are not a subset of "dogs" in the way "pugs" are, and the "except" construction implies a subset relationship.

Datasets

The primary dataset introduced is PreTENS, containing 21,765 artificial sentences. This dataset was programmatically generated from human-verified templates to ensure consistency and allow for systematic testing of various constructions. The argument nouns were drawn from a diverse set of semantic categories.

The PreTENS dataset is a subset of a larger dataset called DuckRabbit. The PreTENS dataset was specifically curated to focus on presuppositional semantic knowledge and was made available in three languages.

Key Insights and Ideas

  • Presuppositions are a Rich Area for NLP Evaluation: The task demonstrates that evaluating presuppositional failures is a valuable way to probe deeper semantic understanding in language models, going beyond traditional syntactic or semantic tasks.
  • World Knowledge is Crucial: Detecting these violations often requires models to access and utilize world knowledge to understand taxonomic relationships (e.g., knowing that "dogs" are a type of "animal").
  • Cross-Lingual Generalization is Challenging: While models were trained and tested across three languages, performance varied, highlighting the difficulties in achieving consistent cross-lingual semantic understanding.
  • Training Data Doesn't Guarantee Performance: Interestingly, some constructions that were not present in the training set were still predicted accurately by top models, suggesting a degree of generalization beyond direct exposure. Conversely, some constructions present in training did not always lead to better performance.
  • Task Nuance Matters: The regression task (Sub-task 2) revealed different performance patterns compared to the binary classification task (Sub-task 1), indicating that models might struggle to capture the fine-grained degree of acceptability.

Key Results

  • Strong Performance on Sub-task 1: The best systems achieved an F1-macro score of 94.49% on the binary classification task.
  • Good Performance on Sub-task 2: The top systems achieved a Spearman correlation coefficient of 0.80 on the regression task.
  • Language Differences: French models generally performed better in the regression task, while English models excelled in the binary classification task.
  • Construction-Specific Performance: The "TYPE" construction was consistently one of the most challenging across languages, while "ANDTOO" and "COMPARATIVES" were generally easier in the binary task. However, in the regression task, "TYPE" and "PARTICULAR" showed lower error rates, while "ANDTOO" and "COMPARATIVES" saw increased errors.
  • Human vs. Model Correlation: A strong correlation was found between human judgments and model predictions, particularly in French and Italian, suggesting that the models are capturing some aspects of human semantic intuition.

Limitations and Future Work

  • Artificial Data: The sentences were artificially generated, which, while allowing for systematic control, might not fully capture the complexity and nuances of natural language.
  • Limited Scope of Presuppositions: The task focused on a specific type of presupposition; future work could explore a wider range of presuppositional phenomena.
  • Analysis of Linguistic Variants: The authors suggest that future work should delve deeper into the linguistic variants of constructions and the semantic categories of arguments to better understand model behavior.
  • Systematicity of Trends: Further analysis is needed to understand the systematicity of why certain constructions are easier or harder for models, especially in relation to their presence or absence in training data.
  • Future Research: The authors hope that the PreTENS dataset will encourage further research into models that can incorporate world knowledge and common sense for a more robust understanding of linguistic properties.

Summary updated

BibTeX
@inproceedings{zamparelli2022semeval,
  author = {Roberto Zamparelli and Shammur Chowdhury and Dominique Brunato and Cristiano Chesi and Felice Dell’Orletta and Md. Arid Hasan and Giulia Venturi},
  title = {{SemEval-2022 Task 3: PreTENS-Evaluating Neural Networks on Presuppositional Semantic Knowledge}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure