How to Handle Different Types of Out-of-Distribution Scenarios in Computational Argumentation? A Comprehensive and Fine-Grained Field Study

Andreas Waldis, Yufang Hou, Iryna Gurevych

TL;DR

This paper presents a comprehensive, fine-grained study on how language models (LMs) handle Out-of-Distribution (OOD) scenarios in Computational Argumentation (CA). It challenges the prevailing assumption that In-Context Learning (ICL) is universally superior for OOD data, demonstrating instead that the optimal training paradigm depends heavily on the specific type of distribution shift (topic, domain, or language). Notably, prompt-based fine-tuning excels during topic shifts, while ICL is better suited for domain shifts with severe label discrepancies.

What Problem Does This Paper Address?

Computational Argumentation (CA) models human reasoning through complex natural language processing tasks like stance detection, argument quality evaluation, and evidence classification. However, creating annotated datasets for CA is incredibly expensive and time-consuming. This data scarcity means that deployed models frequently encounter text from uncovered topics, domains, or languages—known as Out-of-Distribution (OOD) scenarios.

While previous research has evaluated OOD generalization uniformly or focused on a single type of shift in isolation, real-world distribution shifts are highly heterogeneous. This paper addresses the lack of a systematic framework to evaluate and handle different types of covariant distribution shifts (specifically topic, domain, and language shifts) within CA.

Figure 1: Common OOD types of computational ar- gumentation covering evidence classification (Shnarch et al., 2018), and mono or multilingual stance detection (Hardalov et al., 2021; Vamvas and Sennrich, 2020).

Methods

The authors introduce a rigorous evaluation protocol that judges models on three distinct criteria:

  1. Applicability: The absolute task performance, measured via the macro $F_1$ score.
  2. Reliability: How well the model's optimization objective aligns with actual performance (quantified using the Kendall correlation $\tau$ between development loss and task metric).
  3. Stability: The consistency of the model across different random seeds and data folds.

Using this framework, they systematically benchmark three primary learning paradigms across twelve different LMs:

  • Vanilla Fine-Tuning (FT): Training standard task-specific classification heads.
  • Prompt-based Fine-Tuning (P+FT): Formulating the classification task as a cloze test to leverage the LM's pre-trained masked language modeling (MLM) head.
  • In-Context Learning (ICL): Evaluating large language models (LLMs) using task-specific instructions and few-shot demonstrations without updating weights.
  • Parameter-Efficient Fine-Tuning (PEFT): Exploring methods like LoRA, P-Tuning, and Prompt-Tuning to scale gradient-based learning efficiently.

Datasets

The study evaluates models across eleven CA and related classification tasks derived from 13 distinct datasets, categorized by their enforced distribution shifts:

  • Topic Shifts: Argument Quality (arg-qua), Argument Similarity (arg-sim), Argument Classification (arg-cls), and Evidence Classification (evi-cls).
  • Domain Shifts: Sentiment Classification (review), Multi-Dataset Stance Detection (stance), and Multi-Dataset Entailment (entail).
  • Multi-Shift (Language/Topic/Domain): Multilingual Stance Detection (x-stance) and Multilingual Sentiment Classification (x-review).

Key Insights and Ideas

  • No Free Lunch for OOD: There is no single "best" paradigm for OOD generalization. The nature of the shift dictates the solution.
  • Semantic vs. Label Shifts: Prompt-based fine-tuning (P+FT) is highly effective for topic shifts because it naturally handles semantic variations while maintaining stable label distributions. Conversely, ICL shines in domain shifts where heavy discrepancies in label distributions (high KL divergence) cause gradient-based fine-tuning to fail.
  • Pre-training Architecture Dictates P+FT Success: LMs trained purely with token-only objectives (like DeBERTa's masked language modeling) benefit significantly more from prompt-based fine-tuning than models trained with additional sentence-level objectives (like BERT's next-sentence prediction).
  • The Untapped Power of Base Models: Massive parameter scaling isn't always necessary. Smaller, base-sized models (like DeBERTa-v3) can match or outperform giant LLMs on OOD tasks when paired with the right fine-tuning strategy.
Figure 2: Comparison of our study, covering topic (or- ange), domain (blue), or language (square/circle) covari- ant distributions shifts with previous studies that mainly consider single shifts.

Key Results

  • DeBERTa-v3 Dominance: DeBERTa-v3 combined with P+FT outperformed vanilla fine-tuning in 10 out of 11 tasks, even matching in-distribution (ID) performance on argument similarity and sentiment review tasks.
  • P+FT vs. FT: On average across all tasks, DeBERTa-v3 with P+FT achieved an OOD Applicability score of 74.6% compared to 72.3% for vanilla FT, while demonstrating superior Reliability ($\tau = -78.4$ vs. $-72.6$).
  • ICL Strengths: For highly challenging cross-dataset domain shifts (like stance and entail), ICL using smaller reasoning-optimized models like Orca-2 (66.2% average applicability) outperformed larger models like GPT-3.5 (64.9%) and successfully bypassed the generalization flaws of gradient-based methods.
  • LoRA Efficiency: Implementing LoRA ($r=4$) on base models yielded performance on par with full-parameter tuning while offering better stability than scaling up to 3-billion-parameter models (such as T5 or Flan-T5), which proved highly sensitive to random seeds.

Limitations and Future Work

  • Input Length Restrictions: Prompt-based fine-tuning requires wrapping inputs in templates, which consumes valuable token space. While CA tasks in this study featured relatively short texts, this could be a major bottleneck for long-document processing.
  • Inference Efficiency: The paper focuses heavily on training paradigms but does not explore methods to optimize inference speeds or computational costs for larger LLMs in production environments.
  • Data Contamination: The authors note that the massive pre-training corpora of commercial LLMs (like GPT-3.5) likely contain copies of popular evaluation datasets, making it difficult to cleanly separate true OOD generalization from rote memorization.

Summary updated

Abstract

The advent of pre-trained Language Models (LMs) has markedly advanced natural language processing, but their efficacy in out-of-distribution (OOD) scenarios remains a significant challenge. Computational argumentation (CA), modeling human argumentation processes, is a field notably impacted by these challenges because complex annotation schemes and high annotation costs naturally lead to resources barely covering the multiplicity of available text sources and topics. Due to this data scarcity, generalization to data from uncovered covariant distributions is a common challenge for CA tasks like stance detection or argument classification. This work systematically assesses LMs' capabilities for such OOD scenarios. While previous work targets specific OOD types like topic shifts or OOD uniformly, we address three prevalent OOD scenarios in CA: topic shift, domain shift, and language shift. Our findings challenge the previously asserted general superiority of in-context learning (ICL) for OOD. We find that the efficacy of such learning paradigms varies with the type of OOD. Specifically, while ICL excels for domain shifts, prompt-based fine-tuning surpasses for topic shifts. To sum up, we navigate the heterogeneity of OOD scenarios in CA and empirically underscore the potential of base-sized LMs in overcoming these challenges.
BibTeX
@inproceedings{waldis2024how,
  author = {Andreas Waldis and Yufang Hou and Iryna Gurevych},
  title = {{How to Handle Different Types of Out-of-Distribution Scenarios in Computational Argumentation? A Comprehensive and Fine-Grained Field Study}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.795},
}
Figure