Aristoxenus at SemEval-2023 Task 4: A Domain-Adapted Ensemble Approach to the Identification of Human Values behind Arguments

Dimitrios Zaikis, Stefanos D. Stefanidis, Konstantinos Anagnostopoulos, Ioannis VlahavasACL 2023

TL;DR

This paper tackles the challenge of automatically identifying human values within arguments, a crucial task for understanding behavior and communication. The authors propose a Transformer-based system that leverages domain-adapted pre-training and a novel ensemble approach with majority voting to achieve competitive results on the SemEval-2023 ValueEval task.

What Problem Does This Paper Address?

Understanding the underlying human values that drive arguments is essential across various fields, from psychology and sociology to marketing and conflict resolution. However, automatically detecting these values in natural language has been a relatively underexplored area. The SemEval-2023 Task 4 (ValueEval) aimed to address this by developing systems that can classify arguments into one of 20 predefined value categories. This task requires models to go beyond surface-level text analysis and grasp the nuanced motivations and beliefs embedded within argumentative discourse.

Methods

The core of the proposed system, named "Aristoxenus," is built upon the RoBERTa language model. The authors employ a domain-adapted pre-training strategy, specifically a "second-phase pre-training" (also referred to as task-adaptation pre-training or TAPT). This involves further training the RoBERTa model on domain-specific unlabeled data to better align it with the nuances of argumentative language.

A key innovation is the data transformation technique. Instead of directly feeding the premise, stance, and conclusion, they concatenate them into a single sentence, replacing the explicit stance with a connecting phrase that semantically links the premise and conclusion. This aims to create a more continuous and contextually rich input for the model.

For the classification itself, they adopt an One-Versus-All (OVA) strategy. This means training 20 separate binary classifiers, one for each value category, where each classifier learns to distinguish instances of its target category from all other categories. To further enhance robustness, they create an ensemble of these OVA classifiers. This ensemble consists of three distinct sets of models, each trained with different data sampling strategies (original, down-sampled, and a combination of best models) and hyperparameter tuning. The final prediction is determined by a majority vote across these three sets.

Datasets

The primary dataset used is the ValueEval dataset from SemEval-2023 Task 4. This dataset contains arguments, each with a conclusion, a premise, and the premise's stance towards the conclusion. The authors also mention using unlabeled data for the domain-adaptation pre-training phase.

Key Insights and Ideas

  • Task-adaptation is crucial: Fine-tuning pre-trained language models (like RoBERTa) on task-specific data significantly improves performance, indicating that LMs can learn to generate better embeddings by aligning them to the specific task.
  • OVA outperforms single multi-label classifiers: The One-Versus-All approach, where separate binary classifiers are trained for each label, dramatically improves performance compared to a single multi-label classifier. This suggests that handling each value category independently is more effective.
  • Ensembling adds robustness: Combining predictions from multiple models trained with different strategies (data sampling, hyperparameter tuning) through majority voting leads to a more stable and potentially better overall performance.
  • Data transformation matters: Replacing the explicit stance with a semantically rich connecting phrase in the input sentence helps the model better capture the relationship between the premise and conclusion.

Key Results

The proposed Aristoxenus system achieved a macro-F1 score of 0.47 on the official SemEval-2023 ValueEval test set. This performance was competitive, outperforming the baseline BERT approach and the 1-Baseline in 19 out of 20 value categories. While it fell short of the absolute best approach in the competition, it demonstrated the effectiveness of their ensemble and domain-adaptation strategies. The authors also found that the base version of RoBERTa generally performed better than its large counterpart in their experiments.

Limitations and Future Work

The authors acknowledge that their system, while competitive, did not achieve the top performance in the task. They suggest that investigating different language models beyond RoBERTa and exploring alternative data transformation techniques could further enhance predictive capabilities. Additionally, the paper notes that the effectiveness of data sampling techniques varied per label, highlighting the need for label-specific optimization.

Summary updated

BibTeX
@inproceedings{zaikis2023aristoxenus,
  author = {Dimitrios Zaikis and Stefanos D. Stefanidis and Konstantinos Anagnostopoulos and Ioannis Vlahavas},
  title = {{Aristoxenus at SemEval-2023 Task 4: A Domain-Adapted Ensemble Approach to the Identification of Human Values behind Arguments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure