Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023)

Chloé Braud, Yang Janet Liu, Eleni Metheniti, Philippe Muller, Laura Rivière, Attapol Rutherford, Amir ZeldesACL 2023

TL;DR

The DISRPT 2023 shared task brought together researchers to tackle the complex problem of automatically understanding how sentences and clauses relate to each other in text. By providing standardized datasets and evaluation metrics across multiple discourse formalisms and languages, the task aims to advance the development of more robust and generalizable discourse parsing systems. This work is crucial for improving downstream NLP applications that rely on a deep understanding of text structure, such as summarization, question answering, and machine translation.

What Problem Does This Paper Address?

Understanding the "glue" that holds text together – the relationships between different parts of a discourse – is a fundamental challenge in Natural Language Processing (NLP). While humans effortlessly grasp how sentences connect (e.g., cause-effect, contrast, elaboration), machines struggle with this. Different linguistic theories and annotation schemes (like RST, SDRT, PDTB) exist to model these discourse relations, but they often use different units of analysis and labeling conventions. This fragmentation makes it difficult to develop general-purpose discourse parsers that can work across various datasets and languages. The DISRPT (Discourse Relation Parsing and Treebanking) shared task series aims to bridge this gap by promoting flexible methods that can handle diverse guidelines and by fostering the convergence towards common standards. DISRPT 2023, specifically, focuses on three core subtasks: discourse segmentation (identifying the basic units of discourse), connective detection (finding explicit linguistic markers of relations), and relation classification (assigning a label to the identified relation).

Methods

DISRPT 2023, as a shared task, doesn't propose a single novel method but rather evaluates the diverse approaches submitted by participating teams. The core methodology revolves around providing a unified framework for training and evaluating discourse parsing systems. This includes:

  • Standardized Datasets: Training, development, and test datasets were provided in a uniform format, drawing from established corpora like RST, SDRT, PDTB, and DEP (discourse dependencies). This standardization allows for direct comparison of different systems.
  • Multi-formalism and Multi-lingual Approach: The task explicitly supports various discourse formalisms and languages, encouraging the development of methods that are not tied to a single framework or linguistic context.
  • Three Subtasks: The task is broken down into:
  • Discourse Segmentation: Identifying elementary discourse units (EDUs).
  • Connective Detection: Locating explicit discourse connectives (e.g., "because," "however," "therefore").
  • Relation Classification: Assigning a discourse relation label (e.g., Cause, Contrast, Elaboration) between EDUs.
  • Joint Evaluation: The task aims to promote joint evaluation of discourse parsing approaches, suggesting that a holistic understanding of discourse structure is more valuable than tackling each subtask in isolation.

Datasets

DISRPT 2023 utilized existing, well-established discourse corpora, adapted into a uniform format for the shared task. These include:

  • RST (Rhetorical Structure Theory): A widely used framework for analyzing text structure based on rhetorical relations.
  • SDRT (Segmented Discourse Representation Theory): A theory that models discourse structure and meaning representation.
  • PDTB (Penn Discourse Treebank): A large corpus annotated with discourse relations and connectives.
  • DEP (Discourse Dependencies): A framework focusing on dependency relations between discourse units.

The paper emphasizes that these datasets span various languages and genres, providing a rich testbed for developing generalizable discourse parsing methods.

Key Insights and Ideas

  • The Need for Unified Evaluation: The fragmentation of discourse parsing research across different formalisms and datasets is a significant bottleneck. DISRPT 2023 highlights the value of a shared task that standardizes data and evaluation to foster progress.
  • Flexibility is Key: Given the diversity of discourse annotation schemes, systems that can adapt to different guidelines and formalisms are crucial for real-world applicability.
  • Interplay of Subtasks: While the task is divided into segmentation, connective detection, and relation classification, the underlying idea is that these components are interdependent. Progress in one area can inform and improve performance in others.
  • Bridging Theory and Practice: The task serves as a vital bridge between theoretical linguistic models of discourse and practical NLP applications, pushing for more linguistically informed computational models.
  • Future of Discourse NLP: The organizers express hope that future CODI workshops will continue to feature discourse analysis shared tasks, emphasizing the domain's need for more research, diversified evaluation, and expanded coverage of languages and text types.

Key Results

The provided document is a proceedings paper, outlining the shared task itself rather than presenting the results of specific participating systems. Therefore, specific quantitative results from the DISRPT 2023 task are not detailed in this introductory document. The key "result" is the successful organization and execution of the shared task, bringing together researchers and providing a platform for evaluating discourse parsing systems under standardized conditions. The papers presented within these proceedings would contain the specific results of the participating teams.

Limitations and Future Work

  • Limitations: The paper implicitly acknowledges the limitation that different corpora and formalisms have inherent differences that can be challenging to reconcile completely. The need for "flexible methods" suggests that perfect generalization across all frameworks is an ongoing challenge.
  • Future Work: The organizers explicitly state their hope for continued shared tasks on discourse analysis in future CODI workshops. They advocate for:
  • Promoting research that leads to more thorough and diversified evaluation.
  • Working towards more consistent standards in discourse annotation.
  • Expanding the scope to cover more languages and text types not yet well-represented in discourse analysis research.

Summary updated

BibTeX
@inproceedings{braud2023proceedings,
  author = {Chloé Braud and Yang Janet Liu and Eleni Metheniti and Philippe Muller and Laura Rivière and Attapol Rutherford and Amir Zeldes},
  title = {{Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023)}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure