DiscoFlan: Instruction Fine-tuning and Refined Text Generation for Discourse Relation Label Classification

Kaveri AnuranjanaACL 2023

TL;DR

This paper introduces DiscoFlan, a novel approach for discourse relation classification that leverages instruction fine-tuning of large language models (LLMs) like Flan-T5. By treating relation classification as a text generation task and incorporating a refinement step based on label distributions, DiscoFlan achieves strong performance, particularly on low-resource datasets, without relying on hand-crafted features.

What Problem Does This Paper Address?

Discourse Relation Classification (DRC) is a crucial but challenging task in Natural Language Processing (NLP) that involves identifying the semantic relationships between different parts of a text. These relationships, such as cause-effect, contrast, or elaboration, are fundamental to understanding text coherence and meaning. However, DRC is often a low-resource task, meaning there's a scarcity of labeled data, especially across multiple languages and formalisms. Traditional methods often require extensive feature engineering, which is time-consuming and may not generalize well. Furthermore, seq2seq generative models, while powerful, can sometimes produce "mismatched" labels that don't strictly adhere to the predefined label set, especially in low-resource scenarios. This paper aims to address these challenges by developing a robust, multilingual DRC system that minimizes reliance on manual feature engineering and effectively handles label mismatches.

Methods

The core of DiscoFlan lies in two main innovations:

  • Instruction Fine-tuning for Seq2Seq Generation: Instead of traditional classification, DiscoFlan frames DRC as a sequence-to-sequence (seq2seq) generation task. It utilizes Flan-T5, a powerful multilingual LLM pre-trained on instruction-following tasks. The model is fine-tuned using specific prompts that instruct it to generate the discourse relation label given a pair of text segments. This approach leverages the LLM's strong generalization capabilities and allows it to learn from diverse datasets and languages.
  • Refinement Logic for Label Mismatches: To combat the issue of generative models producing labels outside the defined set (mismatches), DiscoFlan introduces a post-processing refinement step. This step involves cleaning up generated labels by removing common affixes and then matching the resulting lemma against the dataset's known labels. If a generated label's lemma doesn't match any in the set, it's replaced with the majority label from the training dataset. This "DiscoFlan+Ref" variant aims to ensure that the final output strictly adheres to the expected label space.

Datasets

The paper evaluates DiscoFlan on the DISRPT 2023 shared task datasets. These datasets cover various languages and formalisms for discourse relation parsing and treebanking. The authors mention that the 2023 task includes 11 new datasets, expanding upon the 15 datasets used by previous state-of-the-art models.

Key Insights and Ideas

  • LLMs as Generative Classifiers: Framing classification tasks as generation problems with LLMs can unlock new capabilities, especially for low-resource scenarios. Instruction fine-tuning proves effective in guiding these models to produce specific, structured outputs like discourse relation labels.
  • The Power of Refinement: Even with powerful LLMs, generative approaches can lead to label mismatches. A simple, distribution-aware refinement step can significantly improve the accuracy and adherence to the target label set, particularly for low-resource datasets where the model might struggle to generalize perfectly.
  • Eliminating Hand-Crafted Features: DiscoFlan demonstrates that it's possible to achieve competitive results in DRC without relying on labor-intensive, hand-crafted features, which was a common practice in earlier DRC systems.

Key Results

DiscoFlan, especially with the refinement logic (DiscoFlan+Ref), shows promising results on the DISRPT 2023 shared task. The paper highlights that DiscoFlan+Ref consistently outperforms the baseline model and, in some cases, approaches state-of-the-art performance. Specifically, the refinement logic is shown to improve performance on low-resource datasets by addressing label mismatches. The authors note that while their model doesn't always achieve the absolute best performance, it offers a strong, feature-less, multilingual alternative.

Limitations and Future Work

The authors acknowledge several limitations:

  • Majority Label Assumption: The refinement logic relies on the assumption that the majority label in the training set is representative. This might not hold for perfectly balanced datasets.
  • Overfitting on Single Labels: For simpler datasets, the model can sometimes overfit to a single label.
  • Model Size Constraints: Due to resource limitations, they used FlanT5-small and did not evaluate FlanT5-large, which could potentially yield better results.

Future work includes exploring more advanced decoding strategies, improving prompts for better label representation, and evaluating larger FlanT5 models. They also suggest that enriching datasets manually or automatically could lead to more representative tasks.

Summary updated

BibTeX
@inproceedings{anuranjana2023discoflan,
  author = {Kaveri Anuranjana},
  title = {{DiscoFlan: Instruction Fine-tuning and Refined Text Generation for Discourse Relation Label Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure