TL;DR
This paper tackles the challenging task of automatically identifying 20 different human values embedded within textual arguments. The authors propose a system that leverages transformer models (BERT and RoBERTa) combined with an undersampling technique to address the significant class imbalance in the dataset, achieving competitive results against a BERT baseline.
What Problem Does This Paper Address?
Understanding the underlying human values that drive arguments is crucial for various applications, from opinion analysis to marketing. However, automatically classifying these values in text is a complex problem. The SemEval-2023 Task 4 (ValueEval) specifically aims to classify arguments into 20 distinct human value categories. A major challenge in this task is the highly imbalanced nature of the dataset, where most instances belong to the "negative" class (i.e., the argument does not draw on a particular value). This imbalance can lead to machine learning models overfitting and performing poorly on minority classes.
Methods
The core of the proposed approach involves using pre-trained transformer models, specifically BERT and RoBERTa, fine-tuned for the argument classification task. The authors utilized the Hugging Face Transformers library for this purpose. To combat the class imbalance, they employed an undersampling strategy. This technique reduces the number of instances in the majority class to better balance the dataset. Additionally, the system was designed to process the argument's conclusion, stance, and premise as a combined input, hypothesizing that this holistic view would provide richer context for value identification compared to using individual components.
Datasets
The primary dataset used in this work is the SemEval-2023 Task 4: Identification of Human Values behind Arguments (ValueEval) dataset. This dataset consists of arguments extracted from social science literature, with each argument comprising an ID, conclusion, stance of the premise towards the conclusion, and the premise text. For each argument, labels indicate the presence (1) or absence (0) of 20 different human value categories. The dataset is noted for its significant class imbalance.
Key Insights and Ideas
- Combined Input is Key: Using the conclusion, stance, and premise together as a single input significantly improved performance over using individual components, highlighting the importance of contextual information for value identification.
- Undersampling for Imbalance: The undersampling technique proved effective in mitigating the negative impact of class imbalance, leading to improved F1 scores, particularly for minority classes.
- Transformer Power: Fine-tuning state-of-the-art transformer models like BERT and RoBERTa demonstrates their strong capability in understanding and extracting nuanced information like human values from text.
- Value Identification is Challenging: Even with advanced models, certain value categories with very few positive instances (e.g., "Stimulation," "Humility," "Face") remain particularly difficult to classify accurately.
Key Results
The proposed approach, combining undersampling and transformer models using the conclusion, stance, and premise as input, outperformed the baseline BERT model. The system achieved an overall F1 score of 0.46, securing the 24th position in the competition. Notably, the performance gap was most pronounced in highly imbalanced classes, where the authors nearly doubled the performance compared to the baseline. For instance, undersampling improved the F1 score for the "Security: societal" label from 0.53 to 0.57 and for "Self-direction: action" by 18%.
Limitations and Future Work
The authors acknowledge that identifying human values in text remains a challenging problem, especially for categories with very few positive examples. They suggest that data augmentation techniques could be explored to increase the size of the training data and potentially improve model performance. Furthermore, extending the system to other languages would require additional preprocessing and potentially language-specific models. Investigating other transformer-based models and their variants is also proposed as a direction for future research.
Summary updated