How to Handle Different Types of Out-of-Distribution Scenarios in Computational Argumentation? A Comprehensive and Fine-Grained Field Study

Andreas Waldis, Yufang Hou, Iryna Gurevych

Introduction: The Challenge of Out-of-Distribution Data

Computational Argumentation (CA) aims to enable artificial intelligence to understand, evaluate, and generate human-like reasoning. However, the field faces a significant bottleneck: data scarcity. Annotating arguments is a complex, expert-level task, often resulting in models trained on narrow datasets—such as debating "nuclear energy"—that are subsequently expected to perform on disparate topics like "healthcare" or different platforms like social media.

When test data deviates from the training distribution, it creates an Out-of-Distribution (OOD) scenario. While many studies treat OOD as a monolithic challenge, this research argues that OOD is heterogeneous. Specifically, it distinguishes between Topic Shift (changes in subject matter), Domain Shift (changes in writing style or source), and Language Shift.

Examples of Topic, Domain, and Language shifts in Computational Argumentation tasks.

Methodology and Learning Paradigms

The authors conducted an extensive field study evaluating 12 language models (LMs) across eight learning paradigms. The primary comparison focused on:

  • Vanilla Fine-Tuning (FT): Standard supervised learning on the target task.
  • Prompt-based Fine-Tuning (P+FT): Framing the task as a natural language "fill-in-the-blank" (cloze) test.
  • In-Context Learning (ICL): Providing few-shot examples to large-scale models (e.g., GPT-3.5, Llama-2) without weight updates.

Performance was measured using a three-pronged framework: Applicability (general $F_1$ score), Reliability (alignment between training loss and test performance), and Stability (consistency across random seeds).

Experimental Setup and Datasets

The study utilized 11 tasks derived from 13 datasets, categorized by shift type:

  • Topic Shift: Includes Argument Quality, Similarity, Classification, and Evidence Classification.
  • Domain Shift: Covers Sentiment analysis, Stance Detection, and Entailment.
  • Language/Mixed Shift: Focuses on Multilingual Stance and Sentiment.

Key Results: The Superiority of Prompting

A central finding is that "one size does not fit all." While In-Context Learning (ICL) is effective for handling label imbalances in domain shifts, it often struggles with deep semantic topic shifts. In contrast, Prompt-based Fine-Tuning (P+FT) emerged as the most robust strategy for topic-heavy tasks.

Performance comparison (Applicability F1) between Vanilla Fine-Tuning (FT) and Prompt-based Fine-Tuning (P+FT) across various language models.

As shown in the figure above, P+FT consistently outperforms vanilla FT across nearly all tested models, including BERT, ALBERT, and DeBERTa-v3. Notably, DeBERTa-v3 paired with P+FT achieved superior results in 10 out of 11 tasks, likely due to its disentangled attention mechanism which better preserves semantic information across neural layers.

Insights and Future Directions

The research highlights a "generalization flaw" in vanilla fine-tuning, where training loss and test performance become uncoupled in OOD scenarios. P+FT mitigates this by maintaining distinct semantic clusters for labels (e.g., Pro vs. Con) deeper into the model's architecture.

For future work, the authors suggest a shift toward "shift-specific" model selection. Practitioners should tailor their training strategy—whether using P+FT for topic shifts or ICL for domain shifts—based on the specific nature of the expected distribution shift. Additionally, parameter-efficient methods like LoRA (Low-Rank Adaptation) offer a promising balance of performance and stability for base-sized models.

Summary updated

Abstract

The advent of pre-trained Language Models (LMs) has markedly advanced natural language processing, but their efficacy in out-of-distribution (OOD) scenarios remains a significant challenge. Computational argumentation (CA), modeling human argumentation processes, is a field notably impacted by these challenges because complex annotation schemes and high annotation costs naturally lead to resources barely covering the multiplicity of available text sources and topics. Due to this data scarcity, generalization to data from uncovered covariant distributions is a common challenge for CA tasks like stance detection or argument classification. This work systematically assesses LMs’ capabilities for such OOD scenarios. While previous work targets specific OOD types like topic shifts or OOD uniformly, we address three prevalent OOD scenarios in CA: topic shift, domain shift, and language shift. Our findings challenge the previously asserted general superiority of in-context learning (ICL) for OOD. We find that the efficacy of such learning paradigms varies with the type of OOD. Specifically, while ICL excels for domain shifts, prompt-based fine-tuning surpasses for topic shifts. To sum up, we navigate the heterogeneity of OOD scenarios in CA and empirically underscore the potential of base-sized LMs in overcoming these challenges.
BibTeX
@inproceedings{waldis2024how,
  author = {Andreas Waldis and Yufang Hou and Iryna Gurevych},
  title = {{How to Handle Different Types of Out-of-Distribution Scenarios in Computational Argumentation? A Comprehensive and Fine-Grained Field Study}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.795},
}
Figure