KaMiKla at SemEval-2022 Task 3: AlBERTo, BERT, and CamemBERT—Be(r)tween Taxonomy Detection and Prediction

Karl Vetter, Miriam Segiet, Klara LennermannNAACL 2022

TL;DR

This paper presents KaMiKla's participation in SemEval-2022 Task 3, which focuses on detecting and predicting taxonomic (is-a) relationships between words. The team leveraged pre-trained BERT models for both classification and regression tasks across English, French, and Italian, achieving respectable but not top-tier results. A key finding is that the specific linguistic constructions used to express these relationships significantly impact model performance, suggesting that future work should consider these constructions more explicitly.

What Problem Does This Paper Address?

Computational linguistics has long been interested in understanding semantic relationships between words, particularly taxonomic relationships (e.g., "dog is a type of animal"). Resources like WordNet and BabelNet have been developed to capture these, but automatically identifying and predicting them in natural language remains a challenge. SemEval-2022 Task 3 specifically aimed to evaluate how well neural networks can:

  1. Classify sentences based on whether they express a valid taxonomic relationship.
  2. Predict the acceptability or strength of a taxonomic relationship on a Likert scale.

The task covered three languages (English, French, and Italian) and involved artificially generated sentences designed to test models' understanding of presuppositions and semantic nuances. The challenge lies not just in recognizing the core "is-a" link but also in handling the diverse ways these relationships are expressed and the potential for ambiguity or misdirection within sentence structures.

Methods

The KaMiKla team employed a standard yet effective approach: using pre-trained transformer-based language models. Specifically, they fine-tuned:

  • BERT base uncased for English.
  • CamemBERT for French.
  • ALBERTO (an Italian BERT model) for Italian.

For both subtasks (binary classification and regression), separate models were trained for each language. The core BERT architecture was extended with a fully connected hidden layer and an output layer.

  • Subtask 1 (Classification): The models were trained using cross-entropy loss with the AdamW optimizer and weight decay. Gradient clipping was used to prevent exploding gradients, and early stopping was implemented based on the validation loss.
  • Subtask 2 (Regression): Similar to subtask 1, but the output layer predicted a single continuous value. The output and targets were normalized to the [0, 1] range. Mean squared error was used as the loss function. To improve robustness, ten models were trained per language, and the final prediction was the median of these models' outputs, as medians are less sensitive to outliers than means.

Hyperparameter tuning was performed using grid search.

Datasets

The paper utilized the datasets provided for SemEval-2022 Task 3.

  • Subtask 1: Consisted of 5,837 training sentences and 14,560 test sentences per language. Sentences were labeled with a binary indicator (1 for acceptable, 0 for unacceptable taxonomic relations).
  • Subtask 2: Involved 524 training sentences and 1,009 test sentences per language. These were annotated with a seven-point Likert scale score representing the acceptability of the taxonomic relation.

The paper notes that after the evaluation phase, additional labels for specific syntactic constructions were released, which the authors used for post-hoc analysis.

Key Insights and Ideas

  • Construction Matters: The most striking finding is how heavily model performance (both classification and regression) varied based on the specific linguistic construction used in the sentence (e.g., "butnot," "except," "generally"). Some constructions were handled well, while others, like "type" and "generally," proved extremely challenging, often resulting in near-zero F1 scores or poor correlations.
  • Language-Specific Performance: While BERT models generally perform well, there were notable differences in performance across languages for specific constructions. For instance, the French classification model outperformed others, while the French regression model struggled with certain constructions.
  • Training Data vs. Test Data Mismatch: The authors observed significant differences in sentence length and construction distribution between the training and test sets. This mismatch, particularly in sentence length and the absence of certain constructions (like "unlike") in the training data, might have hindered the models' ability to generalize effectively.
  • Pre-trained Models are Powerful but Not a Silver Bullet: The use of pre-trained BERT models provided a strong baseline and allowed for good performance on many constructions. However, the task's complexity, especially with specific constructions and limited data for regression, meant that even these powerful models struggled to achieve top results.

Key Results

  • Subtask 1 (Classification): KaMiKla ranked 15th out of 21 groups overall, with an F1 score of 77.38%. Individual language scores were: Italian (0.68 F1), French (0.76 F1), and English (0.73 F1). Performance varied greatly by construction, with F1 scores ranging from near 0 (e.g., "generally") to over 0.9 (e.g., "butnot," "prefer").
  • Subtask 2 (Regression): The team ranked 11th out of 17 groups, achieving a RHO (Spearman correlation) of 0.078. Top-performing models reached correlations around 0.8, indicating a significant gap. Again, performance varied by construction, with some showing positive correlations and others negative.

Limitations and Future Work

  • Data Scarcity: The authors acknowledge that the limited size of the training data, especially for Subtask 2, likely contributed to the models' suboptimal performance and difficulty in generalizing.
  • Training/Test Distribution Mismatch: The observed differences in sentence length and construction distribution between training and test sets are highlighted as a potential issue that might have impacted fine-tuning.
  • Future Directions:
    • Tailoring Fine-tuning: The authors suggest tailoring the fine-tuning process of BERT models to specific linguistic constructions.
  • Construction-Aware Models: Combining BERT with a classifier that explicitly identifies the input sentence's construction type could improve performance.
  • Data Augmentation: Given the data scarcity, exploring data augmentation techniques could be beneficial.

Summary updated

BibTeX
@inproceedings{vetter2022kamikla,
  author = {Karl Vetter and Miriam Segiet and Klara Lennermann},
  title = {{KaMiKla at SemEval-2022 Task 3: AlBERTo, BERT, and CamemBERT—Be(r)tween Taxonomy Detection and Prediction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure