UU-Tax at SemEval-2022 Task 3: Improving the generalizability of language models for taxonomy classification through data augmentation

Injy Sarhan, Pablo Mosteiro, Marco SpruitNAACL 2022

TL;DR

This paper tackles the challenge of making language models generalize better to unseen patterns and noun pairs in taxonomy classification tasks. The authors propose a novel two-stage fine-tuning approach using data augmentation techniques, achieving a strong 3rd place in the SemEval-2022 Task 3.

What Problem Does This Paper Address?

The SemEval-2022 Task 3 (PreTENS) aims to evaluate how well neural language models can identify taxonomic relationships between noun pairs within sentences, specifically when faced with novel sentence patterns or noun combinations not seen during training. This is a crucial problem because real-world applications often require models to handle variations and new data beyond their training distribution. The task presents a significant challenge: the test datasets are much larger than the training sets, and they contain unseen patterns, making it difficult for models to generalize. This directly addresses the broader challenge of improving the robustness and generalizability of deep learning models in Natural Language Processing (NLP).

Methods

The core of the proposed solution, named UU-Tax, lies in a two-stage fine-tuning procedure applied to the ELECTRA language model for sub-task 1 (binary classification).

  • Stage 1: Data Augmentation with NLPAug: The first stage focuses on enriching the training data. The authors utilize the NLPAug library to apply two augmentation techniques:
  • Insertion: Randomly inserting words that fit the local context.
  • Substitution: Replacing words with contextually appropriate alternatives. Crucially, for substitution, they only applied it to sentences labeled as acceptable (label 1) to avoid introducing errors. The number of insertions and substitutions was limited to two per sentence to prevent excessive deviation from the original meaning.

  • Stage 2: Translation-based Augmentation: The second stage involves further fine-tuning using translated versions of the training data. For each language (English, French, Italian), the datasets from the other two languages were translated into the target language. This translation-based augmentation, combined with the original training data, helps the model learn from variations across languages and potentially different phrasing.

For sub-task 2 (regression), a simpler approach was adopted: using the Universal Sentence Encoder (USE) to generate sentence embeddings, followed by various regression models (Linear Regression, K-Nearest Neighbors, Decision Tree, and Support Vector Regressor).

Datasets

The primary datasets used are from the SemEval-2022 Task 3 (PreTENS), which includes sentences in English, French, and Italian.

  • Sub-task 1 (Binary Classification):
    • Training set: 5,838 sentences
    • Test set: 14,556 sentences
  • Sub-task 2 (Regression):
    • Training set: 524 sentences
    • Test set: 1,009 sentences

The paper also mentions using the SemEval-2020 Common Sense Validation task dataset for some experimental explorations, though it was not part of the final UU-Tax model.

Key Insights and Ideas

  • Data Augmentation is Key for Generalization: The experiments strongly suggest that data augmentation is not just helpful but essential for improving the generalizability of language models on tasks with unseen patterns and data. Simply fine-tuning pre-trained models on the limited training data led to poor performance.
  • Two-Stage Fine-Tuning Outperforms Single-Stage: The proposed two-stage approach, combining NLPAug and translation, proved more effective than single-stage fine-tuning methods. This suggests a sequential learning process where initial augmentation helps the model adapt, and subsequent translation-based augmentation further solidifies its understanding.
  • ELECTRA's Strength in Generalization: ELECTRA, with its discriminator-based pre-training, demonstrated superior generalizability compared to BERT and RoBERTa in this specific task, outperforming them significantly in terms of F1 score.
  • Translation as a Powerful Augmentation Tool: Translating data across languages proved to be a highly effective strategy for increasing data variance and improving model robustness.
  • Proper Names as a Source of Error: The error analysis revealed that proper names were a common cause of misclassifications, indicating a potential area for future model improvements.

Key Results

  • Sub-task 1 (Classification): The UU-Tax model achieved a global F1Binary score of 91.25%, securing 3rd place out of 21 teams. This was only a small margin behind the top two teams.
  • Sub-task 2 (Regression): The proposed model achieved a Spearman's rho score of 0.221, ranking 5th place out of 17 teams.
  • Data Augmentation Impact: Ablation studies showed that removing translated data from the second stage led to a significant precision drop (11.42%), highlighting the importance of translation. Similarly, removing NLPAug data also negatively impacted performance.
  • Model Comparison: UU-Tax (two-stage ELECTRA) outperformed other experimented models, including multi-task fine-tuning and data-enriched fine-tuning, by a notable margin (e.g., 3.63% and 4.62% higher F1 scores respectively).

Limitations and Future Work

  • Limited Training Data: The small size of the training datasets, especially for sub-task 2, was a significant limitation, making it difficult to train neural networks directly and hindering the generalization of pre-trained models.
  • Sub-task 2 Regression Challenges: The regression task proved more difficult, with some models performing poorly on specific languages (e.g., French). The authors noted that USE might be better optimized for English.
  • Proper Name Misclassifications: The error analysis identified proper names as a source of misclassification. The authors suggest future work could involve envisioning proper names as hyponyms of adjacent nouns.
  • Adversarial Training: The authors suggest that adversarial training could be a promising avenue for future work to further improve model robustness and generalization across both sub-tasks.

Summary updated

BibTeX
@inproceedings{sarhan2022uu,
  author = {Injy Sarhan and Pablo Mosteiro and Marco Spruit},
  title = {{UU-Tax at SemEval-2022 Task 3: Improving the generalizability of language models for taxonomy classification through data augmentation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure