Cross-Lingual Metaphor Detection for Low-Resource Languages

Anna Hülsing, Sabine Schulte im WaldeNAACL 2024

TL;DR

This paper investigates cross-lingual metaphor detection (MD) for languages with varying data availability, comparing neural and non-neural models. They find that the MAD-X adapter architecture performs best, especially with hyperparameter tuning, and that zero-shot mBERT offers a decent baseline, while few-shot approaches are sensitive to shot selection.

What Problem Does This Paper Address?

Metaphor detection (MD) is crucial for many NLP tasks, but research has largely focused on high-resource languages like English. It's unclear how the amount of pretraining data for large language models (LLMs) impacts MD performance, particularly for low-resource languages. This paper aims to bridge this gap by comparing neural and non-neural cross-lingual MD models across languages with diverse pretraining data levels, including the low-resource language Latin.

Methods

The study compares two main types of models:

  • Neural Models:
    • mBERT (Multilingual BERT): Used in both zero-shot and few-shot settings. For zero-shot, it's fine-tuned on the source language (English) and evaluated on target languages. For few-shot, it undergoes a second fine-tuning stage with a small amount of target language data.
  • MAD-X: An adapter-based architecture designed to adapt LLMs for cross-lingual transfer. It involves training language and task adapters.
  • Non-Neural Model:
    • Random Forest Classifier: This model uses conceptual features like abstractness, imageability, and supersenses, similar to previous work, and is trained on vector space models.

The core task is binary classification: determining if a target word is used metaphorically or literally within its context.

Datasets

The study utilizes the following datasets:

  • Source Language (English):
    • Tsvetkov et al. (2014) dataset (222 instances)
    • Mohammad et al. (2016) dataset (1,639 instances), augmented with the Tsvetkov dataset to a total of 1,861 instances for training.
  • Target Languages:
    • Russian: Tsvetkov et al. (2014) dataset (240 instances).
  • German: Köper and Schulte im Walde (2016) dataset, balanced and reduced to 1,792 instances.
  • Latin: Lexham Figurative Language of the New Testament Dataset (Westbury et al., 2016), with 100 manually annotated sentences.

Key Insights and Ideas

  • MAD-X shines with tuning: While zero-shot mBERT offers a good starting point, MAD-X, especially after hyperparameter tuning, achieves the best performance across most scenarios, highlighting the importance of adaptation techniques for cross-lingual tasks.
  • Few-shot is tricky: Few-shot learning with mBERT proved highly sensitive to the selection of training examples ("shots"). This makes it impractical for cross-lingual settings where target language validation data is unavailable for optimal shot selection.
  • Non-neural models lag behind: The random forest classifier, despite using established conceptual features, was consistently outperformed by the neural models, even in low-resource settings.
  • Hyperparameter tuning is crucial for MAD-X: MAD-X's performance is heavily dependent on suitable hyperparameters, and using a third-language dataset for validation proved effective when target language validation data is absent.
  • German's challenges: The German dataset presented unique difficulties, possibly due to the prevalence of idioms and "computationally challenging" particle verbs, which neural models struggled with compared to Russian.

Key Results

  • Best overall performance: MAD-X achieved the highest F1-scores, particularly for German and Latin when using the basic training dataset with hyperparameter tuning. For Russian, MAD-X with an augmented training dataset and default hyperparameters performed best.
  • Zero-shot mBERT: Consistently outperformed the majority baseline across all target languages, showing decent results even with limited training data.
  • Few-shot mBERT: Performance was inconsistent and highly dependent on shot selection, often falling below other models.
  • Random Forest: Generally performed worse than neural models, though it showed competitive results for Russian.
  • Latin performance: Even with minimal pretraining data, neural models, especially MAD-X, showed promising results for Latin.

Limitations and Future Work

  • Sentence-level focus: The study focused on individual sentences. Future work should extend to longer sequences and authentic data, potentially using sequence-based metaphor detection.
  • Limited language scope: The experiments covered only a subset of languages. Repeating the study with more target languages, especially those outside the Indo-European family, is recommended.
  • English as source: Using only English as the source language might not be optimal. Exploring other high-resource languages as sources could yield better cross-lingual transfer.
  • Joint fine-tuning: The paper suggests exploring joint fine-tuning methods for few-shot learning, as demonstrated by Schmidt et al. (2022), to improve robustness and performance.
  • Comparison with newer LLMs: Future work could compare the performance of the presented models with newer, larger multilingual LLMs like mT5.
  • Idioms and extended metaphors: The qualitative analysis highlighted challenges with idioms and extended metaphors in German. Further investigation into these specific linguistic phenomena is warranted.

Summary updated

BibTeX
@inproceedings{hulsing2024cross,
  author = {Anna Hülsing and Sabine Schulte im Walde},
  title = {{Cross-Lingual Metaphor Detection for Low-Resource Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure