Diversity Over Size: On the Effect of Sample and Topic Sizes for Topic-Dependent Argument Mining Datasets

Benjamin Schiller, Johannes Daxenberger, Andreas Waldis, Iryna Gurevych

Introduction

In the era of Large Language Models (LLMs), the prevailing wisdom in Natural Language Processing (NLP) has often been "bigger is better." However, creating high-quality datasets for specialized tasks like Topic-Dependent Argument Mining (TDAM) remains a significant bottleneck. TDAM requires identifying and classifying arguments relative to specific, often controversial, topics—a task that demands expert-level reasoning to distinguish valid claims from fallacies.

This research challenges the raw scale approach by demonstrating that topic diversity is a far more potent driver of model performance than simple sample volume. By prioritizing a wide range of topics over a high density of samples per topic, researchers can reduce training data requirements by up to 90% while retaining 95% of maximum performance.

Experimental Methodology

To investigate the optimal composition of TDAM datasets, the authors conducted two distinct types of experiments across four major datasets, including the newly introduced FS150T-Corpus, which features 21,600 samples across 150 diverse topics.

  1. Sample Experiments: Models were trained on an increasing number of samples per topic to identify the "saturation point" where performance gains plateau.
  2. Topic Experiments: The total number of training samples was held constant (e.g., 2,880 samples), while the distribution was varied between many samples from a few topics versus few samples from many topics.

Performance was evaluated across supervised models (ERNIE 2.0 and FLAN-T5 XL) and zero-shot LLM baselines (Llama2-70B and ChatGPT).

Efficiency and the "90/95 Rule"

A primary finding of the study is the rapid saturation of performance in supervised models. For the ERNIE 2.0 model, utilizing just 11-15% of available training data was sufficient to reach 95% of the maximum possible F1 macro score.

Figure 3: Sample experiments on the FS150T-Corpus showing rapid performance gains for supervised models compared to LLM baselines.

As shown in the sample experiments, while zero-shot LLMs like Llama2-70B provide a respectable baseline, they are quickly outperformed by smaller, fine-tuned models. Even with as few as 500 to 2,000 diverse samples, a medium-sized model like ERNIE 2.0 can surpass the zero-shot capabilities of much larger counterparts.

Diversity as a Performance Multiplier

The "Topic Experiments" revealed that diversity acts as a multiplier for generalization. When the total sample size is fixed, increasing the number of topics significantly boosts performance. This suggests that exposure to a wide variety of "outlier" topics helps models learn the underlying structure of argumentation better than repetitive exposure to the same few subjects.

Figure 5: Topic experiments on the IBM-Corpus for FLAN-T5 XL, illustrating how increasing topic diversity for a fixed sample size (2,880 samples) leads to dramatic performance improvements.

In some instances, increasing topic diversity while keeping the sample count fixed improved model performance by over 4 percentage points. This effect was particularly pronounced in larger models like FLAN-T5 XL, which, while "hungrier" for data, showed significant gains as the variety of training topics increased.

Economic and Practical Implications

The practical implications for dataset curation are profound. By shifting the focus from volume to diversity, the cost of annotation can be slashed. The authors estimate that a dataset like FS150T, which would cost approximately $3,266 to annotate fully, could achieve comparable utility for just $290 by following a high-diversity, low-sample-count strategy.

Furthermore, models trained on high-diversity corpora like FS150T demonstrated superior generalization, achieving new State-of-the-Art (SOTA) results on the UKP Sentential Argument Mining Corpus, outperforming previous benchmarks by up to 21.6 percentage points in F1 macro for specific topics.

Conclusion

This research provides a clear roadmap for researchers on a budget: prioritize the number of topics over the number of samples per topic. While larger models eventually achieve the highest scores, the strategic selection of diverse data allows even modest hardware and smaller models to achieve competitive, SOTA-level performance in the complex domain of argument mining.

Summary updated

Abstract

The task of Argument Mining, that is extracting and classifying argument components for a specific topic from large document sources, is an inherently difficult task for machine learning models and humans alike, as large Argument Mining datasets are rare and recognition of argument components requires expert knowledge. The task becomes even more difficult if it also involves stance detection of retrieved arguments. In this work, we investigate the effect of Argument Mining dataset composition in few- and zero-shot settings. Our findings show that, while fine-tuning is mandatory to achieve acceptable model performance, using carefully composed training samples and reducing the training sample size by up to almost 90% can still yield 95% of the maximum performance. This gain is consistent across three Argument Mining tasks on three different datasets. We also publish a new dataset for future benchmarking.
BibTeX
@inproceedings{schiller2024diversity,
  author = {Benjamin Schiller and Johannes Daxenberger and Andreas Waldis and Iryna Gurevych},
  title = {{Diversity Over Size: On the Effect of Sample and Topic Sizes for Topic-Dependent Argument Mining Datasets}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.608},
}
Figure