Diversity Over Size: On the Effect of Sample and Topic Sizes for Topic-Dependent Argument Mining Datasets

Benjamin Schiller, Johannes Daxenberger, Andreas Waldis, Iryna Gurevych

TL;DR

This paper demonstrates that for Topic-Dependent Argument Mining (TDAM), diversity of topics is more important than the sheer number of samples. The researchers found that by carefully selecting diverse training topics, they could reduce training data by up to 90% while still maintaining 95% of the model's peak performance, potentially saving thousands of dollars in annotation costs.

What Problem Does This Paper Address?

Argument Mining is the task of automatically identifying and classifying arguments within text. Unlike sentiment analysis, which often relies on surface-level keywords, argument mining requires identifying evidence and reasoning, making it difficult for both machines and non-expert human annotators.

The specific sub-field, Topic-Dependent Argument Mining (TDAM), adds another layer of complexity: the model must determine if an argument is relevant to a specific, given topic (e.g., "Should we ban organic food?"). Because identifying these nuances requires expert knowledge, creating large datasets is incredibly expensive and time-consuming. The authors challenge the "bigger is better" trend in dataset creation, asking instead: What is the optimal composition of a TDAM dataset in terms of the number of topics versus the number of samples per topic?

Methods

The authors conducted a systematic study using two primary model architectures:

  1. ERNIE 2.0: A medium-sized (110M parameters) model pre-trained on word-, structure-, and semantic-aware tasks.
  2. FLAN-T5 XL: A large language model (1.3B parameters) fine-tuned on instruction tasks.
  3. Zero-shot Baselines: Llama2-70B and ChatGPT were used to establish a performance floor without fine-tuning.

They performed two main types of experiments:

  • Sample Experiments: Increasing the total number of training samples while keeping the number of topics constant to find the point of diminishing returns.
  • Topic Experiments: Fixing the total number of samples but increasing the number of distinct topics (diversity sampling) to see if "outlier" topics improve model robustness.

Datasets

The study utilized four datasets, including one new benchmark:

  • FS150T-Corpus (New): Created by the authors to facilitate few-shot learning research. It contains 21,600 samples across 150 controversial topics, with exactly 144 samples per topic.
  • UKP Corpus: A standard TDAM dataset with 8 topics and ~25k samples.
  • IAM-Corpus: Wikipedia-based data focused on claim extraction across 100 topics.
  • IBM-Corpus: A large-scale dataset containing ~30k samples across 221 topics.

Key Insights and Ideas

  • The 95% Threshold: Models reach "acceptable" performance (95% of their maximum possible score) remarkably early. For ERNIE 2.0, this happened using only 11-15% of the available training data.
  • Diversity as Robustness: Increasing the number of topics (even while keeping the total sample count the same) consistently improved performance for medium-sized models. This suggests that exposing the model to a wider variety of "topic embeddings" helps it generalize better to unseen subjects.
  • LLMs vs. Fine-tuning: While zero-shot LLMs (like ChatGPT) perform decently, they are quickly overtaken by even medium-sized models after just 500–2,000 fine-tuning samples.
  • Cost Efficiency: The authors calculated that their proposed "diverse but small" composition could reduce annotation costs for a dataset like FS150T from $3,266 down to just $290 without significant performance loss.

Key Results

The researchers found that the "Diversity Over Size" approach holds true across different domains and tasks:

  • Data Reduction: On the FS150T-Corpus, ERNIE 2.0 reached 95% of its max performance with only 1,920 samples (11% of the data). FLAN-T5 XL required more data (33%) but still allowed for a 67% reduction in dataset size.
  • Generalization: Models trained on the diverse FS150T-Corpus (many topics, few samples each) outperformed models trained on the UKP Corpus (few topics, many samples each) when tested on both datasets.
  • State-of-the-Art: By using their diversity-focused training approach, the authors achieved new state-of-the-art results on the UKP Corpus, outperforming much larger models like TACAM-BERT Large.
Figure 1: Sample experiments on the FS150T-Corpus
Topic experiments on the IBM-Corpus showing that for a fixed number of samples (e.g., 960), increasing the number of topics generally improves F1 macro scores.

Limitations and Future Work

  • Task Specificity: The findings are currently limited to TDAM. While the authors speculate these insights apply to Stance Detection or Question Answering, they haven't tested them yet.
  • Model Instability: Larger models like FLAN-T5 XL showed more unstable performance at very low sample sizes compared to the medium-sized ERNIE 2.0.
  • Upper Bounds: The experiments capped at ~22k samples and 160 topics. It remains unclear if performance would eventually saturate or decline if topic diversity were increased even further (e.g., to thousands of topics).
  • Inference Costs: While LLMs provide high peak performance, the authors note that ERNIE 2.0 is significantly faster and cheaper to run in production (111 samples/sec vs 34 samples/sec for FLAN-T5).

Summary updated

Abstract

The task of Argument Mining, that is extracting and classifying argument components for a specific topic from large document sources, is an inherently difficult task for machine learning models and humans alike, as large Argument Mining datasets are rare and recognition of argument components requires expert knowledge. The task becomes even more difficult if it also involves stance detection of retrieved arguments. In this work, we investigate the effect of Argument Mining dataset composition in few- and zero-shot settings. Our findings show that, while fine-tuning is mandatory to achieve acceptable model performance, using carefully composed training samples and reducing the training sample size by up to almost 90% can still yield 95% of the maximum performance. This gain is consistent across three Argument Mining tasks on three different datasets. We also publish a new dataset for future benchmarking.
BibTeX
@inproceedings{schiller2024diversity,
  author = {Benjamin Schiller and Johannes Daxenberger and Andreas Waldis and Iryna Gurevych},
  title = {{Diversity Over Size: On the Effect of Sample and Topic Sizes for Topic-Dependent Argument Mining Datasets}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.608},
}
Figure