DisCut and DiscReT: MELODI at DISRPT 2023

Eleni Metheniti, Chloé Braud, Philippe Muller, Laura RivièreACL 2023

TL;DR

The MELODI team presents their approach to the DISRPT 2023 shared task on discourse parsing, tackling segmentation, connective identification, and relation classification. They achieve state-of-the-art results in segmentation by leveraging a multilingual transformer architecture and explore novel techniques like adapters and label merging for relation classification, aiming for robust generalization across diverse corpora.

What Problem Does This Paper Address?

Discourse analysis aims to understand how sentences and larger text units are structured and related to form coherent meaning. This involves identifying elementary discourse units (EDUs), recognizing explicit discourse connectives (like "because" or "however"), and classifying the semantic or pragmatic relations between these units (e.g., "concession," "evidence"). However, existing discourse parsing frameworks (like RST, SDRT, and PDTB) vary significantly, leading to diverse annotation guidelines and relation sets across corpora. This heterogeneity makes it challenging to build robust, generalizable models that perform well across different languages, domains, and theoretical frameworks. The DISRPT 2023 shared task aims to address this by providing a unified benchmark and encouraging research into multilingual and cross-framework approaches.

Methods

The MELODI team developed two systems: DisCut for segmentation and connective identification, and DiscReT for relation classification.

For DisCut, they employed a multilingual transformer architecture, specifically XLM-ROBERTa large. They experimented with freezing lower layers of the transformer to create lighter models and improve performance. For segmentation and connective identification, they treated these as sequence labeling tasks, using a BIO scheme.

For DiscReT, they also utilized a transformer-based approach, focusing on multilingual models to handle diverse languages. To address the challenge of a large and varied set of relation labels across corpora, they implemented two key strategies:

  1. Label Merging and Lowercasing: They standardized labels by converting them to lowercase and manually merging similar or variant labels to reduce the label space.
  2. Adapters: They incorporated adapters, a lightweight fine-tuning technique that adds small, trainable modules to pre-trained transformer layers. This allows for efficient transfer learning and parameter sharing across tasks and languages. They also experimented with freezing adapter layers and varying training epochs.

Datasets

The DISRPT 2023 shared task provided a comprehensive benchmark with 26 corpora across 13 languages and 4 theoretical frameworks (PDTB, RST, SDRT, and dependency-based). This included 10 new corpora for the 2023 edition, featuring surprise datasets and out-of-domain (OOD) data for English, Portuguese, and Turkish (TED talks). The authors used these diverse datasets for training and evaluation, aiming to build models that generalize well.

Key Insights and Ideas

  • Multilingualism is Key: The success of their models, particularly for segmentation and connective identification, strongly relies on using multilingual pre-trained transformers like XLM-ROBERTa. This allows them to leverage knowledge from high-resource languages to improve performance on low-resource ones.
  • Layer Freezing for Efficiency and Performance: Freezing lower layers of the transformer models not only made them lighter but also, in many cases, improved performance, suggesting that these layers capture generalizable linguistic features.
  • Adapters for Relation Classification: Adapters proved to be an effective and lightweight solution for handling the complexity of discourse relation classification across multiple corpora and languages, offering a promising avenue for efficient transfer learning.
  • Label Normalization is Crucial: The significant effort in unifying and merging relation labels was essential for building a single, generalizable model for relation classification, despite the inherent difficulty of the task.
  • Sentence Segmentation is a Bottleneck: The performance of their segmentation and connective identification systems was highly dependent on the quality of sentence segmentation, especially for the "Plain" track where gold sentence boundaries were not provided. Using tools like Trankit showed significant improvements.

Key Results

  • Segmentation: The MELODI team achieved state-of-the-art results for discourse segmentation on the treebanked track, outperforming other systems.
  • Connective Identification: They ranked second for connective identification on the treebanked track and were the only system to submit results for the plain track, achieving higher performance than the winner of DISRPT 2021.
  • Relation Classification: While their relation classification system's performance was lower than some specialized approaches, it was the only system attempting to mix all corpora, suggesting better generalization capabilities. They presented a final submission composed of three models, balancing performance and interpretability.
  • General Improvements: Compared to DISRPT 2021, they observed general improvements across tasks, though a notable drop in mean performance for connective detection was attributed to the inclusion of many new corpora and OOD datasets.

Limitations and Future Work

  • Implementation Issues: The authors acknowledge that implementation issues, particularly with converting labels back to their original format for evaluation, significantly impacted their reported test results for relation classification.
  • Label Space Reduction: While they reduced the number of unique labels, the label space for relation classification remains large and complex, posing a continuous challenge.
  • Domain Shift: Performance on OOD datasets (like TEDm) showed a drop, highlighting the challenge of domain shift even when using models trained on similar languages and frameworks.
  • Future Work: The team plans to further explore joint training strategies for lower-resource languages, investigate domain transfer techniques, and examine the transferability of their framework to new languages and domains. They also aim to tackle implementation issues and improve their multilingual approach.

Summary updated

BibTeX
@inproceedings{metheniti2023discut,
  author = {Eleni Metheniti and Chloé Braud and Philippe Muller and Laura Rivière},
  title = {{DisCut and DiscReT: MELODI at DISRPT 2023}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure