Abstract
The advent of pre-trained Language Models (LMs) has markedly advanced natural language processing, but their efficacy in out-of-distribution (OOD) scenarios remains a significant challenge. Computational argumentation (CA), modeling human argumentation processes, is a field notably impacted by these challenges because complex annotation schemes and high annotation costs naturally lead to resources barely covering the multiplicity of available text sources and topics. Due to this data scarcity, generalization to data from uncovered covariant distributions is a common challenge for CA tasks like stance detection or argument classification. This work systematically assesses LMs' capabilities for such OOD scenarios. While previous work targets specific OOD types like topic shifts or OOD uniformly, we address three prevalent OOD scenarios in CA: topic shift, domain shift, and language shift. Our findings challenge the previously asserted general superiority of in-context learning (ICL) for OOD. We find that the efficacy of such learning paradigms varies with the type of OOD. Specifically, while ICL excels for domain shifts, prompt-based fine-tuning surpasses for topic shifts. To sum up, we navigate the heterogeneity of OOD scenarios in CA and empirically underscore the potential of base-sized LMs in overcoming these challenges.
Introduction: The Heterogeneity of OOD in Argumentation
Computational Argumentation (CA) involves complex tasks such as stance detection, argument quality evaluation, and evidence classification. However, the field is plagued by data scarcity due to high annotation costs. Consequently, models trained on specific datasets often struggle when deployed in real-world scenarios where the data distribution differs from the training set.
While "Out-of-Distribution" (OOD) is often treated as a monolithic challenge, this study argues that OOD scenarios in CA are fundamentally heterogeneous. The authors identify three distinct types of covariant shifts:
- Topic Shift: Changes in the subject matter (e.g., from "Solar Energy" to "Universal Healthcare").
- Domain Shift: Changes in the source or style of text (e.g., from social media to formal debating platforms).
- Language Shift: Changes in the language of the discourse (e.g., from German to French).
Experimental Framework
The researchers conducted an extensive field study evaluating 12 different Language Models (ranging from base-sized models like BERT to large-scale models like GPT-3.5) across eight learning paradigms. Key paradigms included:
- Vanilla Fine-Tuning (FT): Standard weight updates for classification.
- Prompt-based Fine-Tuning (P+FT): Reformulating tasks as cloze tests to leverage the model's pre-training knowledge.
- In-Context Learning (ICL): Few-shot prompting without weight updates, typical for Large Language Models (LLMs).
- Parameter-Efficient Tuning (LoRA): Updating only a small subset of parameters.
To provide a holistic view of performance, the study utilized three metrics: Applicability (average $F_1$ score), Reliability (alignment between training loss and performance), and Stability (variance across seeds).
Key Results and Insights
The "No Free Lunch" Theorem for OOD
A major finding of this work is that there is no single best learning paradigm for all OOD scenarios. While recent trends favor In-Context Learning (ICL) for OOD tasks, this study reveals that Prompt-based Fine-Tuning (P+FT) often outperforms ICL, particularly in scenarios involving semantic topic shifts. ICL only tends to excel when there is a significant discrepancy in label distributions, typical of domain shifts.
The Superiority of DeBERTa-v3
Among the models tested, DeBERTa-v3 emerged as a top performer. When using P+FT, it achieved a 3.3-point higher Applicability and 2.9-point better Reliability than RoBERTa. This success is attributed to its disentangled attention mechanism and "token-only" pre-training objective, which help preserve semantic information in the final layers.
The Power of Base Models
One of the most significant takeaways is that base-sized models (86M–300M parameters), when paired with P+FT or LoRA, can match or even exceed the performance of much larger models (13B+ parameters) on CA tasks. This suggests that for many specialized argumentation tasks, massive scale is not a prerequisite for OOD robustness.
Conclusion
This study underscores the importance of a fine-grained approach to OOD challenges in Computational Argumentation. By recognizing that different types of shifts—topic, domain, and language—require different learning strategies, researchers can better navigate the data-scarce landscape of CA. Future work will explore more complex label mapping (verbalizers) and address the stability issues inherent in larger models to further bridge the gap between in-distribution and out-of-distribution performance.
Summary updated