ALBERT-BiLSTM for Sequential Metaphor Detection

Shuqun Li, Jingjie Zeng, Jinhui Zhang, Tao Peng, Liang Yang, Hongfei LinACL 2020

TL;DR

This paper introduces an ALBERT-BiLSTM-softmax model for detecting metaphorical words in sentences, achieving state-of-the-art results on the TOEFL dataset. The authors explore various data preprocessing techniques and model configurations, ultimately finding that using the first subword token from ALBERT's BPE segmentation and a weighted softmax layer yields the best performance.

What Problem Does This Paper Address?

Metaphors are a fundamental aspect of human language, allowing us to express complex ideas and emotions vividly. However, automatically detecting metaphorical language, particularly identifying the specific words that carry the metaphorical meaning, is a challenging task. Traditional methods often struggle with the vastness of metaphorical expressions and the ambiguity between literal and figurative language. This paper tackles the problem of sequential metaphor detection, aiming to classify each word in a sentence as either metaphorical or not, with a specific focus on identifying metaphorical words and verbs.

Methods

The core of the proposed method is an ALBERT-BiLSTM-softmax architecture. ALBERT, a pre-trained language model, is used to extract contextualized word embeddings. A Bidirectional Long Short-Term Memory (BiLSTM) network then processes these embeddings to capture sequential dependencies. Finally, a softmax layer predicts the probability of each word being metaphorical.

Key methodological contributions and explorations include:

  • Data Preprocessing: The authors experimented with three input formats: single sentences, sentence pairs (to capture context), and sentence-prompt pairs (using provided prompts). They found that single sentences performed best, suggesting that metaphors are often localized within the sentence itself.
  • ALBERT Input Representation: To address the issue of ALBERT's BPE tokenization creating longer output sequences than input, they explored three strategies for representing words: using only the first subword token, using the first embedding of each word, or merging subword embeddings. The "first subword token" approach yielded the best results, likely due to its balance of semantic information and reduced input noise for the model.
  • Classification Layer: The paper compares Conditional Random Fields (CRF) with softmax for the final classification. Softmax with class weights outperformed CRF, suggesting that the relationships between metaphorical words and their context might not be as strictly sequential as CRF assumes.
  • Tricks for Improvement:
    • Semantic Merge: They experimented with combining representations from different ALBERT layers, finding that using only the higher layers (more focused on semantics) was more effective than including lower layers (more focused on syntax).
  • Loss with Class Weight: To address the imbalance of metaphorical words (which are relatively rare), they introduced class weights in the loss function, finding a weight of 2 for positive samples to be optimal.

Datasets

The primary dataset used for training and evaluation is the TOEFL dataset. This dataset consists of 180 articles from TOEFL tests, divided into training and testing sets. The paper also mentions testing their model on the VUA dataset, which is another benchmark for metaphor detection.

Key Insights and Ideas

  • Localization of Metaphor: The finding that single sentences perform best for metaphor detection suggests that the immediate context within a sentence is often sufficient to identify metaphorical words, rather than relying heavily on broader discourse context or explicit prompts.
  • Subword Tokenization Strategy: The success of using only the first subword token from ALBERT's BPE segmentation highlights the trade-off between capturing rich semantic information and managing model complexity, especially on smaller datasets. The first subword often carries the core meaning.
  • Softmax vs. CRF: The superiority of softmax over CRF for this task indicates that the "hard" sequential constraints of CRF might not be ideal for metaphor detection, where metaphorical meaning can be more fluid and context-dependent.
  • Class Imbalance is Crucial: The significant improvement gained from using class weights in the loss function underscores the importance of addressing data imbalance in specialized NLP tasks like metaphor detection.

Key Results

The proposed ALBERT-BiLSTM-softmax model achieved competitive results on the TOEFL dataset:

  • TOEFL Dataset:
    • All POS subtask: 0.707 F1-score
    • Verb subtask: 0.728 F1-score
  • VUA Dataset:
    • All POS subtask: 0.712 F1-score
    • Verb subtask: 0.755 F1-score

These results demonstrate the effectiveness of their proposed architecture and optimization strategies.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the TOEFL dataset is relatively small, which can make models susceptible to input noise and affect training.
  • Spelling Errors: The presence of spelling errors in the TOEFL dataset, likely due to its non-native English speaker origin, posed a challenge. While they attempted spell correction, it did not improve performance, suggesting that models might be robust to some level of noise or that the correction process itself introduced issues.
  • Future Work: The paper implicitly suggests that further exploration of pre-trained models and advanced contextual representations could lead to even better metaphor detection performance. Investigating more sophisticated methods for handling data imbalance and exploring cross-lingual metaphor detection could also be fruitful avenues.

Summary updated

BibTeX
@inproceedings{li2020albert,
  author = {Shuqun Li and Jingjie Zeng and Jinhui Zhang and Tao Peng and Liang Yang and Hongfei Lin},
  title = {{ALBERT-BiLSTM for Sequential Metaphor Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure