TL;DR
This paper investigates cross-lingual metaphor detection (MD) for languages with varying data availability, comparing neural and non-neural models. They find that the MAD-X adapter architecture performs best, especially with hyperparameter tuning, and that zero-shot mBERT offers a decent baseline, while few-shot approaches are sensitive to shot selection.
What Problem Does This Paper Address?
Metaphor detection (MD) is crucial for many NLP tasks, but research has largely focused on high-resource languages like English. It's unclear how the amount of pretraining data for large language models (LLMs) impacts MD performance, particularly for low-resource languages. This paper aims to bridge this gap by comparing neural and non-neural cross-lingual MD models across languages with diverse pretraining data levels, including the low-resource language Latin.
Methods
The study compares two main types of models:
- Neural Models:
- mBERT (Multilingual BERT): Used in both zero-shot and few-shot settings. For zero-shot, it's fine-tuned on the source language (English) and evaluated on target languages. For few-shot, it undergoes a second fine-tuning stage with a small amount of target language data.
- MAD-X: An adapter-based architecture designed to adapt LLMs for cross-lingual transfer. It involves training language and task adapters.
- Non-Neural Model:
- Random Forest Classifier: This model uses conceptual features like abstractness, imageability, and supersenses, similar to previous work, and is trained on vector space models.
The core task is binary classification: determining if a target word is used metaphorically or literally within its context.
Datasets
The study utilizes the following datasets:
- Source Language (English):
- Tsvetkov et al. (2014) dataset (222 instances)
- Mohammad et al. (2016) dataset (1,639 instances), augmented with the Tsvetkov dataset to a total of 1,861 instances for training.
- Target Languages:
- Russian: Tsvetkov et al. (2014) dataset (240 instances).
- German: Köper and Schulte im Walde (2016) dataset, balanced and reduced to 1,792 instances.
- Latin: Lexham Figurative Language of the New Testament Dataset (Westbury et al., 2016), with 100 manually annotated sentences.
Key Insights and Ideas
- MAD-X shines with tuning: While zero-shot mBERT offers a good starting point, MAD-X, especially after hyperparameter tuning, achieves the best performance across most scenarios, highlighting the importance of adaptation techniques for cross-lingual tasks.
- Few-shot is tricky: Few-shot learning with mBERT proved highly sensitive to the selection of training examples ("shots"). This makes it impractical for cross-lingual settings where target language validation data is unavailable for optimal shot selection.
- Non-neural models lag behind: The random forest classifier, despite using established conceptual features, was consistently outperformed by the neural models, even in low-resource settings.
- Hyperparameter tuning is crucial for MAD-X: MAD-X's performance is heavily dependent on suitable hyperparameters, and using a third-language dataset for validation proved effective when target language validation data is absent.
- German's challenges: The German dataset presented unique difficulties, possibly due to the prevalence of idioms and "computationally challenging" particle verbs, which neural models struggled with compared to Russian.
Key Results
- Best overall performance: MAD-X achieved the highest F1-scores, particularly for German and Latin when using the basic training dataset with hyperparameter tuning. For Russian, MAD-X with an augmented training dataset and default hyperparameters performed best.
- Zero-shot mBERT: Consistently outperformed the majority baseline across all target languages, showing decent results even with limited training data.
- Few-shot mBERT: Performance was inconsistent and highly dependent on shot selection, often falling below other models.
- Random Forest: Generally performed worse than neural models, though it showed competitive results for Russian.
- Latin performance: Even with minimal pretraining data, neural models, especially MAD-X, showed promising results for Latin.
Limitations and Future Work
- Sentence-level focus: The study focused on individual sentences. Future work should extend to longer sequences and authentic data, potentially using sequence-based metaphor detection.
- Limited language scope: The experiments covered only a subset of languages. Repeating the study with more target languages, especially those outside the Indo-European family, is recommended.
- English as source: Using only English as the source language might not be optimal. Exploring other high-resource languages as sources could yield better cross-lingual transfer.
- Joint fine-tuning: The paper suggests exploring joint fine-tuning methods for few-shot learning, as demonstrated by Schmidt et al. (2022), to improve robustness and performance.
- Comparison with newer LLMs: Future work could compare the performance of the presented models with newer, larger multilingual LLMs like mT5.
- Idioms and extended metaphors: The qualitative analysis highlighted challenges with idioms and extended metaphors in German. Further investigation into these specific linguistic phenomena is warranted.
Summary updated