Unpacking Ambiguous Structure: A Dataset for Ambiguous Implicit Discourse Relations for English and Egyptian Arabic

Ahmed Ruby, Sara Stymne, Christian HardmeierACL 2023

TL;DR

This paper introduces a novel dataset designed to study ambiguous implicit discourse relations, specifically causal and concessive ones, in English and Egyptian Arabic. The research highlights that while humans can readily disambiguate these relations with context, current language models struggle significantly, suggesting a gap in their understanding of nuanced discourse.

What Problem Does This Paper Address?

Effective communication relies heavily on coherence, which is often achieved through discourse relations – the logical connections between parts of a text. While explicit connectives like "so" or "but" make these relations clear, implicit relations, inferred without such markers, are more challenging. Ambiguity arises when a single sentence pair could plausibly represent multiple discourse relations. This paper tackles the problem of how to systematically create and study these ambiguous implicit relations, particularly focusing on the causal vs. concessive distinction, and investigates whether context can help humans and machines resolve this ambiguity.

Methods

The authors propose a set of principles for constructing ambiguous implicit discourse relations. Key principles include ensuring the relation is implicit (no overt connectives), that the second argument (Arg2) provides supplementary information for the first (Arg1), and that Arg1 is neutral and context-dependent. They then applied these principles to create a dataset of sentence pairs where the same Arg1 and Arg2 could be interpreted as either causal or concessive, depending on the preceding context. This dataset was validated by human participants and then used to evaluate the performance of language models (BERT and ELECTRA) in identifying the correct implicit connective.

Datasets

The paper introduces a new dataset of ambiguous implicit discourse relations for both English and Egyptian Arabic. This dataset consists of sentence pairs designed to be ambiguous between causal and concessive readings, each paired with two distinct preceding contexts that force one reading over the other. The dataset was validated by human participants and is intended to facilitate future research on prosody's role in disambiguation.

Key Insights and Ideas

  • Human vs. Machine Discrepancy: Humans excel at using context to disambiguate implicit discourse relations, even when the linguistic cues are minimal. Language models, however, struggle significantly with this task, indicating a gap in their ability to capture nuanced contextual influence.
  • Context is King (for Humans): The study demonstrates that providing specific preceding context is crucial for humans to correctly infer the intended discourse relation, effectively resolving ambiguity.
  • Model Limitations: Current large language models, despite their advancements, fail to leverage context effectively for disambiguating implicit discourse relations. Their performance is poor, often predicting non-connective words, suggesting a fundamental challenge in their discourse understanding capabilities.
  • Dataset Construction Principles: The paper offers a valuable framework for creating controlled ambiguous discourse relation examples, which can be a useful resource for future NLP research in this area.

Key Results

  • Human validation showed high agreement among participants in identifying the intended implicit connectives, especially after iterative refinement of the dataset.
  • Language models (BERT and ELECTRA) performed poorly on the task of filling in ambiguous implicit connectives. For Arabic examples, the models' performance was "extremely poor," with many predicted words not functioning as connectives.
  • The models showed minimal differences in their predictions between the "so" (causal) and "but" (concessive) groups, suggesting they did not effectively capture the contextual influence that humans relied on.

Limitations and Future Work

The authors acknowledge that the English and Arabic datasets are not entirely equivalent due to translation nuances. They also note that the language models' poor performance indicates a substantial challenge for current architectures in capturing context for implicit discourse relations. Future work includes conducting controlled experiments on the impact of prosody for disambiguation and constructing more examples to build a classification model for identifying implicit discourse relations.

Summary updated

BibTeX
@inproceedings{ruby2023unpacking,
  author = {Ahmed Ruby and Sara Stymne and Christian Hardmeier},
  title = {{Unpacking Ambiguous Structure: A Dataset for Ambiguous Implicit Discourse Relations for English and Egyptian Arabic}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure