Introduction
In the era of Large Language Models (LLMs), the prevailing wisdom in Natural Language Processing (NLP) has often been "bigger is better." However, creating high-quality datasets for specialized tasks like Topic-Dependent Argument Mining (TDAM) remains a significant bottleneck. TDAM requires identifying and classifying arguments relative to specific, often controversial, topics—a task that demands expert-level reasoning to distinguish valid claims from fallacies.
This research challenges the raw scale approach by demonstrating that topic diversity is a far more potent driver of model performance than simple sample volume. By prioritizing a wide range of topics over a high density of samples per topic, researchers can reduce training data requirements by up to 90% while retaining 95% of maximum performance.
Experimental Methodology
To investigate the optimal composition of TDAM datasets, the authors conducted two distinct types of experiments across four major datasets, including the newly introduced FS150T-Corpus, which features 21,600 samples across 150 diverse topics.
- Sample Experiments: Models were trained on an increasing number of samples per topic to identify the "saturation point" where performance gains plateau.
- Topic Experiments: The total number of training samples was held constant (e.g., 2,880 samples), while the distribution was varied between many samples from a few topics versus few samples from many topics.
Performance was evaluated across supervised models (ERNIE 2.0 and FLAN-T5 XL) and zero-shot LLM baselines (Llama2-70B and ChatGPT).
Efficiency and the "90/95 Rule"
A primary finding of the study is the rapid saturation of performance in supervised models. For the ERNIE 2.0 model, utilizing just 11-15% of available training data was sufficient to reach 95% of the maximum possible F1 macro score.
As shown in the sample experiments, while zero-shot LLMs like Llama2-70B provide a respectable baseline, they are quickly outperformed by smaller, fine-tuned models. Even with as few as 500 to 2,000 diverse samples, a medium-sized model like ERNIE 2.0 can surpass the zero-shot capabilities of much larger counterparts.
Diversity as a Performance Multiplier
The "Topic Experiments" revealed that diversity acts as a multiplier for generalization. When the total sample size is fixed, increasing the number of topics significantly boosts performance. This suggests that exposure to a wide variety of "outlier" topics helps models learn the underlying structure of argumentation better than repetitive exposure to the same few subjects.
In some instances, increasing topic diversity while keeping the sample count fixed improved model performance by over 4 percentage points. This effect was particularly pronounced in larger models like FLAN-T5 XL, which, while "hungrier" for data, showed significant gains as the variety of training topics increased.
Economic and Practical Implications
The practical implications for dataset curation are profound. By shifting the focus from volume to diversity, the cost of annotation can be slashed. The authors estimate that a dataset like FS150T, which would cost approximately $3,266 to annotate fully, could achieve comparable utility for just $290 by following a high-diversity, low-sample-count strategy.
Furthermore, models trained on high-diversity corpora like FS150T demonstrated superior generalization, achieving new State-of-the-Art (SOTA) results on the UKP Sentential Argument Mining Corpus, outperforming previous benchmarks by up to 21.6 percentage points in F1 macro for specific topics.
Conclusion
This research provides a clear roadmap for researchers on a budget: prioritize the number of topics over the number of samples per topic. While larger models eventually achieve the highest scores, the strategic selection of diverse data allows even modest hardware and smaller models to achieve competitive, SOTA-level performance in the complex domain of argument mining.
Summary updated