Turiya at DialAM-2024: Inference Anchoring Theory Based LLM Parsers

Sougata Saha, Rohini SrihariACL 2024

TL;DR

This paper introduces computational models for parsing dialogues using the Inference Anchoring Theory (IAT) framework, which represents dialogues as argument graphs. They compare traditional biaffine parsers with Large Language Models (LLMs), finding that finetuned LLMs significantly improve dialogue argument mining, especially when combined with classification-based approaches.

What Problem Does This Paper Address?

Argument mining, the task of identifying argumentative structures in text, has been extensively studied for monologues. However, formal models for parsing dialogues, which are inherently more complex due to turn-taking and inter-speaker dependencies, are lacking. The DialAM-2024 shared task aims to address this by using the Inference Anchoring Theory (IAT) framework to model argumentation and dialogue information jointly. IAT represents dialogues as graphs where "L-nodes" are speaker turns, "I-nodes" are propositions derived from L-nodes, and various edges connect these nodes to represent argumentative relationships. The challenge is to build computational models that can effectively construct these IAT-based argument graphs from dialogue data.

Methods

The authors propose and compare two main approaches for implementing the IAT framework:

  • Classification-Based Model: This approach uses a Roberta-based classifier with two biaffine layers. These layers predict relationships and their types between nodes (e.g., S-nodes connecting propositions, YA-nodes connecting various elements). This is a standard technique in dependency parsing and relation extraction.
  • Generative Model (LLMs): They leverage Mistral-7B-Instruct-v0.2, a large language model, by framing dialogue argument parsing as a generation task. The LLM is provided with IAT definitions, L-nodes, I-nodes, and TA-nodes, and tasked with sequentially identifying S-nodes and YA-nodes. They experiment with two types of instructions: Type 1, which is more open-ended, and Type 2, which explicitly lists relation labels. To address LLMs sometimes producing incomplete annotations, they implement an iterative decoding approach, re-prompting the model with partially generated outputs.

Datasets

The dataset used for training and evaluation is the QT30 corpus, which contains 1,477 IAT-annotated conversations. The dataset was preprocessed to strictly adhere to IAT relationship definitions. It was split into 1,327 training and 150 testing examples.

Key Insights and Ideas

  • LLMs are viable for dialogue argument parsing: The paper demonstrates that generative LLMs can be effectively finetuned for the nuanced task of constructing IAT-based argument graphs from dialogues, outperforming traditional classification-based methods.
  • Iterative decoding improves LLM completeness: The proposed iterative decoding strategy for LLMs, while computationally more expensive, significantly improves the completeness of generated annotations, ensuring more relationship types are identified.
  • Ensembling yields the best performance: Combining the strengths of both classification-based (biaffine) and generative (LLM) approaches through ensembling leads to the best overall performance, particularly at the "Focused" evaluation level.
  • Type 1 LLM instructions are more effective: LLMs trained with Type 1 instructions (more open-ended) generally outperform those trained with Type 2 instructions (explicitly listing labels), suggesting LLMs can infer relationships more effectively when not overly constrained.

Key Results

  • LLM vs. Biaffine: LLM-based models achieved higher F1 scores than the biaffine classifier across both "General" and "Focused" evaluation levels.
  • Ensemble Performance: The ensemble of the biaffine parser and the Type 1 instruction-based LLM achieved the best Precision, Recall, and F1 scores at the "Focused" level on an internal test set.
  • Official Test Set: On the official test set, their implementation (an ensemble of biaffine and Type 1 LLM) achieved the best F1 score at the "General" level, outperforming majority-based and Roberta-based baselines for both Task A (propositional relations) and Task B (illocutionary relations). Their model performed better on Task B than Task A.

Limitations and Future Work

  • Computational Cost: The iterative decoding approach for LLMs is computationally expensive, approximately 4x to 15x more so than regular decoding.
  • Dataset Size and Scope: While the QT30 corpus is valuable, the authors note that training on a reduced set of relations (due to strict adherence to IAT definitions) might have impacted performance.
  • General vs. Focused Evaluation: The paper highlights that higher "General" scores compared to "Focused" scores indicate the models might be over-relying on non-related nodes. The iterative decoding approach, by prompting for all relationship types, might promote recall but could lead to spurious predictions.
  • Future Work: The paper implicitly suggests further exploration into optimizing LLM prompting strategies and potentially larger, more diverse datasets for dialogue argument mining. The success of ensembling also points towards further research in hybrid approaches.

Summary updated

BibTeX
@inproceedings{saha2024turiya,
  author = {Sougata Saha and Rohini Srihari},
  title = {{Turiya at DialAM-2024: Inference Anchoring Theory Based LLM Parsers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure