StFX-NLP at SemEval-2023 Task 4: Unsupervised and Supervised Approaches to Detecting Human Values in Arguments

Ethan Heavey, Milton King, James HughesACL 2023

TL;DR

This paper explores both supervised and unsupervised machine learning approaches for identifying human values within textual arguments, a task crucial for understanding motivations and fostering productive discussions. While their best-performing ensemble model achieved a respectable 34th place out of 39 teams in the SemEval-2023 competition, the study highlights the significant challenges posed by class imbalance in the dataset.

What Problem Does This Paper Address?

Understanding the underlying human values that drive arguments is a complex but important problem. This is particularly relevant in fields like computational linguistics and social science, where identifying these values can lead to more insightful conversations, better conflict resolution, and a deeper understanding of human motivations. SemEval-2023 Task 4 specifically aimed to classify textual arguments into one of 20 human value categories derived from Schwartz's value continuum. The challenge lies in accurately mapping nuanced textual expressions to these abstract value categories, especially when the dataset exhibits a significant imbalance in the representation of different values.

Methods

The authors experimented with a variety of machine learning models, including both supervised and unsupervised techniques.

  • Supervised Approaches:
    • XGBoost Classifier: A gradient boosting algorithm that builds decision trees sequentially. They employed a two-step process: first, training on BERT embeddings of premises, and second, incorporating TF-IDF scores to refine predictions.
  • Ensemble Models: Two ensemble models were developed using scikit-learn's VotingClassifier.
  • BERT Ensemble: Utilized BERT embeddings ([CLS] token) from premises.
  • Sentence Ensemble: Employed sentence embeddings from the all-MiniLM-L6-v2 model.
  • Both ensemble models were evaluated using "soft voting," which emphasizes predicted probabilities, proving more effective than "hard voting" on the imbalanced dataset.
  • Unsupervised Approach:
    • Threshold Comparison: This method leveraged sentence embeddings of class descriptions (derived from the knowledge resource) and premise embeddings. It calculated cosine similarity between premise and description embeddings, assigning a label if the similarity exceeded a predefined threshold (tuned to 0.2). The class descriptions were carefully curated, with the best performance achieved by combining specific phrases (phrase 1 and phrase 3b) from the definitions.

Datasets

The primary dataset used was the ValueEval'23 dataset (Mirzakhmedova et al., 2023). This dataset comprises approximately 9,000 English records, with the majority sourced from IBM's argument quality dataset, supplemented by data from the Conference for the Future of Europe and group discussion ideas. The data was split into training (60%), validation (20%), and testing (20%) sets.

Key Insights and Ideas

  • Class Imbalance is a Major Hurdle: The study strongly suggests that class imbalance significantly impacts model performance. Classes with very low representation (< 2%) showed substantially lower F₁-scores, indicating that models struggle to learn patterns for these underrepresented values.
  • Soft Voting Outperforms Hard Voting: In the context of imbalanced datasets, "soft" voting in ensemble models proved more effective than "hard" voting. This is because soft voting considers the probability distribution of predictions, giving more weight to less frequent but potentially correct classifications.
  • Curated Class Descriptions Matter for Unsupervised Learning: For the unsupervised threshold comparison model, the specific selection of phrases from the human value definitions was critical. Using a combination of the class name and a more detailed aspect of its representation (phrases 1 and 3b) yielded the best results, highlighting the importance of well-defined and distinct class descriptions.
  • Ensemble Methods Show Promise: The ensemble models, particularly the sentence-embedding-based one, demonstrated the best performance among the submitted approaches, suggesting that combining multiple models can lead to more robust predictions.

Key Results

  • The best performing model was the Ensemble-Sentence model, which achieved a rank of 77th out of 111 submitted runs for the task.
  • The authors' overall team performance ranked 34th out of 39 teams.
  • Classes with high representation (> 11%) had significantly higher average F₁-scores (e.g., 0.565 for Security: personal, 0.5775 for Universalism: concern) compared to low-representation classes (average F₁-score of 0.1175 for Stimulation, 0.13 for Hedonism, and 0.095 for Conformity: interpersonal).
  • The sentence ensemble model performed 7.67% better than the BERT ensemble model during validation.
  • The unsupervised Threshold Comparison model achieved a 94th/111 rank.

Limitations and Future Work

  • Class Imbalance: The most significant limitation acknowledged is the class imbalance in the dataset, which hinders the models' ability to generalize to less frequent classes.
  • Limited Data Representation: The models primarily focused on the "Premise" column of the dataset.
  • English-Only Dataset: The dataset being exclusively in English limits the generalizability of the models to other languages.

Future work suggested includes:

  • Exploring the unsupervised learning approach further by interpreting classes and data in different ways.
  • Incorporating other data columns like "Conclusion" and "Stance" into supervised models for potentially more accurate premise labeling.
  • Investigating state-of-the-art embedding methods and comparing their performance.

Summary updated

BibTeX
@inproceedings{heavey2023stfx,
  author = {Ethan Heavey and Milton King and James Hughes},
  title = {{StFX-NLP at SemEval-2023 Task 4: Unsupervised and Supervised Approaches to Detecting Human Values in Arguments}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure