DFKI-MLST at DialAM-2024 Shared Task: System Description

Arne Binder, Tatiana Anikina, Leonhard Hennig, Simon OstermannACL 2024

TL;DR

This paper presents a novel unified approach for identifying argumentative and illocutionary relations in dialogues, achieving top results in the DialAM-2024 shared task. By framing the problem as n-ary relation classification and employing a robust preprocessing pipeline, the DFKI-MLST system effectively models complex dialogue structures.

What Problem Does This Paper Address?

Dialogue argument mining, the task of identifying and classifying argumentative structures within conversations, is crucial for understanding how people reason and persuade each other. Traditional approaches often focus on binary relations between text spans. However, real-world dialogues are more complex, involving multiple participants, turn-taking, and various types of relationships between propositions (argumentative relations) and between speech acts and propositions (illocutionary relations). The DialAM 2024 shared task specifically addresses this by using the Inference Anchoring Theory (IAT) framework, which models dialogue argumentation as a bipartite graph. This framework requires identifying not only argumentative relations (S nodes) between propositions (I nodes) but also illocutionary relations (YA nodes) that anchor these propositions to dialogue turns (L nodes) and transitions (TA nodes). The challenge lies in handling these diverse relation types, their n-ary nature, and the imbalanced distribution of relation labels within a unified system.

Methods

The DFKI-MLST system tackles all three subtasks of the DialAM shared task by framing them as a single n-ary relation classification problem.

Key methodological components include:

  • Unified Architecture: A single deep learning model is trained to handle all relation types (argumentative and illocutionary).
  • Nodeset Normalization: A crucial preprocessing step that reconstructs a complete graph representation of the dialogue's argumentative structure from the raw data. This involves:
  • Aligning Information (I) nodes with Locution (L) nodes based on textual similarity.
  • Constructing argumentative (S) nodes by mirroring relations from transition (TA) nodes, with special handling for "Inference" relations that might have reversed directions.
  • Assigning gold labels to newly constructed relations.
  • Cleaning the data by removing isolated nodes, self-loops, and invalid argument combinations.
  • Encoding as Relations: The normalized nodesets are converted into n-ary relations. Locution (L) nodes form the base text, and relation labels are prefixed with task identifiers (S, YA:L-to-I, YA:TA-to-S).
  • Deep Learning Model: A contextual text encoder (specifically DeBERTa-v3) is used, augmented with role-specific begin- and end-marker tokens inserted into the base text for relation classification. The model is trained using cross-entropy loss and the Adam optimizer.

Datasets

The primary dataset used is the DialAM dataset, derived from the QT30 corpus of BBC's Question Time episodes. This dataset contains transcriptions of dialogues annotated according to the Inference Anchoring Theory (IAT) framework, representing argumentative and illocutionary relations. The paper mentions splitting this dataset into training (1259 nodesets) and validation (140 nodesets) partitions for model selection.

Key Insights and Ideas

  • Unified Approach Pays Off: Treating all relation types (argumentative and illocutionary) as a single n-ary classification task allows the model to leverage shared information and achieve better overall performance, especially in the "GLOBAL" setting which considers the entire argument map.
  • Preprocessing is Paramount: The nodeset normalization pipeline is critical. It effectively reconstructs the complex graph structures and addresses data inconsistencies, significantly impacting performance. The paper shows that even a "preprocessing only" step (without full normalization) yields substantial improvements.
  • Transformer Encoders are Powerful: DeBERTa-v3 emerges as a strong text encoder, outperforming other models like BART, Mistral, and Llama for this task, particularly in the global settings.
  • Data Imbalance is a Challenge: The DialAM dataset exhibits significant class imbalance, especially for YA relations. While weighted loss and restricting to frequent classes were explored, they didn't consistently improve performance over the baseline, suggesting that rare labels, even with low support, are still valuable.
  • Textual Nuances Matter: Combining both "information" (I) node texts and "locution" (L) node texts for encoding relations improved illocutionary relation identification and overall GLOBAL scores, highlighting the importance of richer contextual information.

Key Results

  • The DFKI-MLST system achieved the best scores in the global setting of the DialAM 2024 shared task.
  • Specifically, they achieved 48.25 F1 at the GLOBAL-Focused level and 67.05 F1 at the GLOBAL-General level.
  • The system also showed strong performance on the ARI task, outperforming other models by +9.11% F1 in ARI-General.
  • DeBERTa-v3 was identified as the best-performing text encoder, yielding improvements of +0.85% F1 in GLOBAL-General and +0.5% F1 in GLOBAL-Focused compared to the second-best model on the validation set.
  • Combining L and I node texts for encoding improved illocution identification and GLOBAL scores by up to 3.4% F1 (Focused) and 1.9% F1 (General) for DeBERTa-v3.

Limitations and Future Work

  • Data Augmentation Nuances: While data augmentation (paraphrasing and EDA) showed some improvement for ARI tasks, it led to worse performance on ILO tasks, suggesting a trade-off or that augmentation strategies need further refinement for different relation types.
  • Class Imbalance: The significant class imbalance remains a challenge. The paper explored weighted loss and restricting to frequent classes, but these did not yield consistent improvements, indicating that more advanced techniques might be needed.
  • Model Comparison: While DeBERTa-v3 excelled in global settings, BART performed better on specific ILO tasks. This suggests that different models might be better suited for different aspects of dialogue argumentation.
  • Future Work: The authors suggest exploring more sophisticated data augmentation techniques, investigating methods to better handle class imbalance, and further analyzing the strengths of different base models for specific relation types. They also mention the potential of combining L and I node texts for richer representations.

Summary updated

BibTeX
@inproceedings{binder2024dfki,
  author = {Arne Binder and Tatiana Anikina and Leonhard Hennig and Simon Ostermann},
  title = {{DFKI-MLST at DialAM-2024 Shared Task: System Description}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure