Lauri Ingman at SemEval-2023 Task 4: A Chain Classifier for Identifying Human Values behind Arguments

Spencer Paulissen, Caroline WendtACL 2023

TL;DR

This paper introduces ValueEval, a novel chain classifier architecture for identifying 20 distinct human values within arguments. Their model significantly outperforms existing baselines by effectively capturing inter-label dependencies, demonstrating a promising new approach to understanding the underlying motivations in textual discourse.

What Problem Does This Paper Address?

Understanding the human values that underpin arguments and opinions is a complex but crucial NLP challenge. These values are often implicit, variable, and interconnected, making them difficult to detect in natural language. Online discourse, in particular, offers a rich but challenging landscape for analyzing these underlying principles. The SemEval-2023 Task 4, ValueEval, aims to address this by tasking systems with classifying whether an argument is associated with any of 20 predefined human values. This is particularly relevant for understanding controversial topics with social, economic, and political implications.

Methods

The authors propose a chain classifier architecture built upon a pre-trained BERT model. The core idea is to leverage BERT's contextualized embeddings for the argument's conclusion, stance, and premise. These embeddings are then fed into a "chain" of classifiers. Each classifier in the chain is responsible for predicting one of the 20 human values. Crucially, each subsequent classifier in the chain receives not only the initial BERT embeddings but also the predictions from all preceding classifiers. This design allows the model to learn and exploit inter-label dependencies, a significant advantage over treating each value classification independently. They experimented with various architectural choices, including unfreezing BERT layers and adjusting the number of hidden layers between BERT and the classifier chain. For hyperparameter tuning, they focused on dropout probability, hidden size, batch size, learning rate, and importantly, the classification threshold, which they found to be a critical factor for performance.

Datasets

The primary dataset used for training and evaluation is the "Argument dataset" adapted from previous work by Kiesel et al. (2022). This dataset contains 5220 unique natural language arguments, each comprising a conclusion, stance, and premise. Arguments are annotated with a binary vector indicating the presence of 20 human value categories. The authors split this dataset into an 80% training set and a 20% development set, ensuring proportional balance of human value categories within each split.

Key Insights and Ideas

  • Inter-label dependencies are key: The paper strongly suggests that human values are not independent. Their chain classifier's success hinges on its ability to learn these relationships, a feature lacking in simpler baseline models.
  • Threshold tuning matters: The authors found that optimizing the classification threshold, rather than sticking to a default 0.5, significantly improved performance by leveraging label density and class distribution.
  • Model size vs. feasibility: While larger models and more layers generally improved performance, computational constraints limited their exploration. This highlights a common trade-off in NLP research.
  • BERT as a strong foundation: The use of pre-trained BERT embeddings proved effective, providing a robust starting point for the downstream classification task.

Key Results

The authors' model achieved a micro F1-score of 0.67 on the Level 1 task (all 20 labels), a substantial improvement over the best baseline model (BERT with an F1-score of 0.25). This demonstrates a significant leap in performance for identifying human values in arguments. The paper also notes that their model trains quickly, requiring only 20-30 minutes per session, which is a considerable advantage for development and deployment.

Limitations and Future Work

The authors acknowledge that model size was a significant limitation due to computational resource constraints. They hypothesize that a larger model could potentially achieve even better results. They also suggest exploring other pre-trained language models like GPT or ALBERT for potentially improved performance, though this would come at the cost of increased training time and complexity. Future work could also involve exploring more advanced multi-label classification methods, such as ensemble methods, to further exploit inter-class dependencies. Data augmentation is another promising avenue, especially given the relatively small size of the training dataset.

Summary updated

BibTeX
@inproceedings{paulissen2023lauri,
  author = {Spencer Paulissen and Caroline Wendt},
  title = {{Lauri Ingman at SemEval-2023 Task 4: A Chain Classifier for Identifying Human Values behind Arguments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure