Noam Chomsky at SemEval-2023 Task 4: Hierarchical Similarity-aware Model for Human Value Detection

Sumire Honda, Sebastian WilharmACL 2023

TL;DR

This paper introduces a novel "hierarchical similarity-aware" model for detecting human values behind arguments. By incorporating semantic similarity scores between arguments and value categories, their approach significantly improves upon a baseline BERT model, particularly for underperforming value categories. This work highlights the effectiveness of leveraging hierarchical dataset structures and semantic similarity for nuanced text classification tasks.

What Problem Does This Paper Address?

Understanding the underlying human values that motivate arguments is crucial for various applications, from analyzing public discourse to building more empathetic AI systems. While previous work has explored moral sentiment detection, directly identifying human values behind arguments has been a less explored area. The challenge lies in the abstract nature of human values and the hierarchical structure of value taxonomies, where broad categories encompass more specific ones. Existing models often struggle to capture these nuances, especially for less represented or more abstract value categories. This paper tackles the SemEval-2023 Task 4, which aims to classify arguments based on a hierarchical taxonomy of human values.

Methods

The core of the proposed method is a "hierarchical similarity-aware" approach built upon a standard BERT embedding model. The authors first establish a baseline using BERT for premise classification into 20 human value categories. Their novel contribution lies in augmenting this baseline by incorporating similarity scores.

Specifically, they utilize Sentence-BERT (SBERT) to calculate the cosine similarity between argument premises and example sentences or descriptions of the human value categories. To handle the hierarchical nature of the dataset, they concatenate related level 1 value labels or their example sentences into single strings before computing similarity. This concatenated string is then compared to the premise using SBERT. The resulting similarity score, ranging from -1 to 1, is then concatenated with the BERT embedding of the premise before being fed into the classification head. This allows the model to leverage both the direct semantic representation of the argument and its similarity to predefined value concepts.

Datasets

The primary dataset used is the Touché23-ValueEval Dataset (Mirzakhmedova et al., 2023), which contains 8865 arguments annotated with human values. This dataset features a hierarchical taxonomy with 54 level 1 values and 20 level 2 categories. The authors followed the provided train/validation/test split (5393:1896:1935). Additionally, they used the supplemental Nahj al-Balagha dataset for testing. Example sentences for level 1 labels were also utilized to enhance the similarity calculation.

Key Insights and Ideas

  • Hierarchical Structure Matters: The paper demonstrates that explicitly leveraging the hierarchical structure of the human value taxonomy, by concatenating related lower-level labels or examples, can improve performance.
  • Similarity as Auxiliary Information: While a model solely based on similarity scores struggled to perform well, integrating these scores as auxiliary information alongside BERT embeddings significantly boosted performance, especially for challenging categories.
  • SBERT's Efficacy for Argument Similarity: The study confirms SBERT's capability in capturing semantic similarity between arguments and value descriptions, even when dealing with concatenated strings.
  • Targeted Improvement: The similarity-aware approach showed particular strength in improving scores for value categories that the baseline BERT model performed poorly on, suggesting it helps in disambiguating or identifying more abstract concepts.

Key Results

The proposed "noam-chomsky" model achieved a macro-average F₁-score of 0.47 on the SemEval-2023 competition. While this score placed them in the middle of the pack (rank 19 out of 39 teams), it represented a sizeable improvement over the BERT baseline. Notably, categories like "Stimulation" and "Humility" saw their scores more than doubled compared to the baseline. The "Security: personal" category, which was the best performing for the BERT baseline, showed a slight decrease in performance. On the validation set, the final model achieved a macro-average F₁-score of 0.44, outperforming both the BERT-only model (0.40) and a similarity-only model (0.20).

Limitations and Future Work

The authors acknowledge that their model, while improved, still falls short of the top-performing approaches. A key limitation is that the similarity score alone, without the BERT embeddings, struggled to perform the classification task effectively, indicating its role as auxiliary information rather than a standalone solution. The paper also notes the challenging aspect of SBERT accurately calculating similarity when one input is a standard sentence and the other is a concatenated string with special tokens.

Future work could explore more sophisticated ways to integrate similarity scores, potentially investigating different concatenation strategies or weighting schemes. Further research into how to best represent and utilize the hierarchical nature of complex taxonomies in NLP tasks remains an open area. The authors also suggest that sentence similarity could be beneficial in other domains where deeper semantic understanding beyond surface-level words is required.

Summary updated

BibTeX
@inproceedings{honda2023noam,
  author = {Sumire Honda and Sebastian Wilharm},
  title = {{Noam Chomsky at SemEval-2023 Task 4: Hierarchical Similarity-aware Model for Human Value Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure