The DISRPT 2023 Shared Task on Elementary Discourse Unit Segmentation, Connective Detection, and Relation Classification

Chloé Braud, Yang Janet Liu, Eleni Metheniti, Philippe Muller, Laura Rivière, Attapol Rutherford, Amir ZeldesACL 2023

TL;DR

The DISRPT 2023 shared task aimed to advance the understanding and automated parsing of discourse structure across various languages and formalisms. This year's iteration introduced new corpora and languages, pushing the boundaries of generalization for discourse parsing systems and highlighting persistent challenges, particularly in implicit relation classification and cross-lingual transfer.

What Problem Does This Paper Address?

Discourse parsing, the task of uncovering the underlying structure of text and how its parts relate to each other, is crucial for many NLP applications like summarization, question answering, and sentiment analysis. However, progress has been hampered by the diversity of theoretical frameworks (e.g., RST, SDRT, PDTB), languages, and domains, making it difficult to evaluate and compare systems. The DISRPT shared task series aims to address this by providing a unified benchmark for key discourse parsing subtasks: identifying elementary discourse units (EDUs), detecting discourse connectives, and classifying the relations between EDUs. DISRPT 2023 specifically sought to test system generalizability across a wider range of data, including new languages and out-of-domain datasets, and to evaluate performance on both treebanked (gold sentence splits) and plain (automatically segmented) data.

Methods

The DISRPT 2023 shared task focused on three core tasks:

  1. Discourse Unit Segmentation (EDU Segmentation): Identifying the boundaries of elementary discourse units (EDUs).
  2. Discourse Connective Detection: Identifying specific words or phrases (connectives) that signal discourse relations.
  3. Discourse Relation Classification: Assigning a semantic label (e.g., "explanation," "contrast") to the relationship between two EDUs.

Participating systems primarily employed transformer-based language models (e.g., BERT, RoBERTa, XLM-ROBERTa, Flan-T5). Some systems used traditional sequence labeling approaches with LSTMs and CRFs, while others leveraged generative models or adversarial training strategies to improve robustness and generalization. The task also included two tracks: "Treebanked" (using gold sentence splits) and "Plain" (relying on automatic sentence splitting), to assess the impact of segmentation errors.

Datasets

DISRPT 2023 featured a comprehensive benchmark comprising 26 corpora across 13 languages. This included:

  • New corpora from the DEP framework.
  • Data for two new languages: Thai and Italian.
  • 3 discourse treebanks annotated in the discourse dependency representation.
  • 6 "surprise" datasets, including out-of-domain (OOD) corpora for which no training data was provided.

These datasets cover various frameworks (RST, SDRT, PDTB, DEP) and domains (news, wiki, scientific documents, conversations).

Key Insights and Ideas

  • Generalization remains a challenge: Despite advances in transformer models, performance drops significantly when systems are applied to unseen genres or domains, especially with limited training data.
  • Sentence splitting matters: The "Plain" track, which relies on automatic sentence segmentation, reveals the crucial role of accurate sentence splitting for downstream discourse parsing tasks. Errors in sentence segmentation can lead to substantial performance degradation.
  • Implicit relations are hard: The gap between performance on explicit (connective-triggered) and implicit (connective-less) discourse relations is substantial, highlighting the difficulty of inferring relations without explicit cues.
  • Cross-lingual transfer is promising but limited: While multilingual models show promise, the performance on new languages or low-resource settings still lags behind well-resourced languages like English.
  • Data scarcity drives innovative merging: Teams explored various strategies for merging data from different corpora to overcome data scarcity, demonstrating the importance of joint learning approaches.

Key Results

  • EDU Segmentation: The MELODI team's DisCut system achieved a mean F1 score of 91.87 on the Treebanked track, outperforming previous winning systems. Scores remained high even on the Plain track, indicating robustness.
  • Connective Detection: The HITS team achieved the highest mean F1 score of 80.47. This task proved challenging with new corpora, especially OOD datasets from TED talks and dialogues, with mean scores around 80%, a notable drop from previous years.
  • Relation Classification: The HITS team won this task with a mean accuracy of 62.36%. Performance varied greatly across corpora, with some challenging datasets (German PCC, Turkish TDB, Dutch NLDT) showing accuracy below 52%. Implicit relations consistently performed much worse (40s) than explicit relations (around 85%).

Limitations and Future Work

  • Data Imbalance and Label Space Explosion: The large number of relation labels (163 unique names) across corpora poses a significant challenge for joint learning and analysis. The paper suggests that efforts to converge on a more standardized set of labels are needed.
  • Impact of Annotation Differences: Discrepancies in annotation guidelines and granularity across corpora make direct comparison and generalization difficult.
  • Future Work: The authors encourage further research using the DISRPT data to develop more robust and generalizable discourse parsing systems. They highlight the need for better understanding of how sentence segmentation errors impact connective detection and for continued efforts to improve implicit relation classification and cross-lingual transfer. The development of more cohesive annotation schemes and the exploration of novel model architectures are also suggested.

Summary updated

BibTeX
@inproceedings{braud2023disrpt,
  author = {Chloé Braud and Yang Janet Liu and Eleni Metheniti and Philippe Muller and Laura Rivière and Attapol Rutherford and Amir Zeldes},
  title = {{The DISRPT 2023 Shared Task on Elementary Discourse Unit Segmentation, Connective Detection, and Relation Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure