TeamEC at SemEval-2023 Task 4: Transformers vs. Low-Resource Dictionaries, Expert Dictionary vs. Learned Dictionary

Nicolas Stefanovitch, Bertrand De Longueville, Mario ScharfbilligACL 2023

TL;DR

This paper explores the effectiveness of transformer models versus dictionary-based approaches for classifying human values in text. Surprisingly, transformers performed on par with very small dictionaries, and only slightly outperformed dictionary methods in a coarse-grained setting, suggesting that for this specific task and dataset, simpler methods can be surprisingly competitive.

What Problem Does This Paper Address?

The paper tackles the challenge of automatically identifying human values expressed in textual arguments. This is crucial for understanding the underlying values that influence public policy and its acceptance. While advanced transformer models are state-of-the-art for many NLP tasks, their effectiveness compared to simpler, more interpretable dictionary-based methods in the nuanced domain of human values remains an open question. The authors also investigate the trade-offs between dictionaries curated by human experts and those automatically learned from data.

Methods

The researchers employed two primary approaches:

  • Dictionary-Based: They built dictionaries by extracting keywords from the training data. This involved lemmatization, stop-word removal, and TF-IDF weighting to identify the most important words for each value class. They experimented with both expert-designed dictionaries and dictionaries learned automatically from the data. Classification was done using simple keyword-matching heuristics.
  • Transformer-Based: They fine-tuned a pre-trained RoBERTa model on the task's training data. Hyperparameter optimization focused on the number of training steps, as initial experiments showed unexpected behavior.

Datasets

The primary dataset used was the ValueEval shared task dataset (SemEval-2023 Task 4). This dataset was constructed from existing sources to cover a wide range of cultural sensitivities, focusing solely on English text. The authors also mention using a previously developed expert dictionary from (Ponizovskiy et al., 2020) for comparison.

Key Insights and Ideas

  • Transformers vs. Small Dictionaries: The most surprising finding is that transformer models performed comparably to dictionaries containing fewer than 1,000 words, even when evaluated on a fine-grained set of 19 categories. This challenges the assumption that complex models are always necessary for high performance.
  • Coarse vs. Fine-Grained: Transformers only showed a clear advantage over dictionary-based approaches in a coarser setting with 10 categories. This suggests that the complexity of distinguishing between many similar values might be a bottleneck for both methods on this dataset.
  • Expert vs. Learned Dictionaries: Expert-designed dictionaries showed comparable precision to learned dictionaries but significantly lower recall. This could indicate that expert dictionaries might be more focused but miss broader mentions, or that the learned dictionaries might be overfit to the specific topics in the dataset.
  • Dataset Homogeneity: The authors hypothesize that the homogeneous language and sentence structure of the dataset might contribute to the relatively small performance gap between sophisticated transformers and simpler dictionary methods. This implies that models highly tuned to this dataset might not generalize well to more diverse text.

Key Results

  • The best transformer model (Roberta 4500) achieved a micro F1 score of 0.458.
  • Dictionary-based approaches performed competitively, with some achieving F1 scores close to the transformer models. For instance, a learned dictionary achieved a micro F1 of 0.57 when evaluated on a coarse-grained setting.
  • In the coarse-grained experiment (10 classes), the transformer model had a micro F1 score 11 points higher than the best dictionary-based approach (Dict 100).
  • Despite the transformer's overall performance, the Dict 100 system showed higher recall than the transformer in some instances.

Limitations and Future Work

  • Overfitting Concerns: The authors express concern about potential overfitting of the transformer models to the specific dataset, especially given the high number of training steps required for optimal performance.
  • Dataset Specificity: The homogeneous nature of the ValueEval dataset is noted as a potential factor influencing the results. The authors suggest that models highly optimized for this dataset might not perform well on more varied text.
  • Future Research: The paper calls for further research to investigate the correlation between the number of labels used and the consistency of their application, as well as to explore how these models generalize to different datasets. They also suggest that the difficulty in interpreting the learned dictionaries for practitioners is a limitation.

Summary updated

BibTeX
@inproceedings{stefanovitch2023teamec,
  author = {Nicolas Stefanovitch and Bertrand De Longueville and Mario Scharfbillig},
  title = {{TeamEC at SemEval-2023 Task 4: Transformers vs. Low-Resource Dictionaries, Expert Dictionary vs. Learned Dictionary}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure