Predicting Human Metaphor Paraphrase Judgments with Deep Neural Networks

Yuri Bizzoni, Shalom LappinNAACL 2018

TL;DR

This paper introduces a new dataset and a deep neural network model for predicting how humans judge metaphor paraphrases. Their model achieves promising results in both classifying paraphrase candidates and ranking them by acceptability, demonstrating the potential of deep learning for understanding figurative language.

What Problem Does This Paper Address?

Understanding and processing metaphor is a significant challenge in Natural Language Processing (NLP). While metaphor detection has seen some progress, the task of metaphor paraphrasing – finding a literal sentence that accurately captures the meaning of a metaphorical expression – has received less attention. This is partly due to a lack of annotated data that specifically targets this problem. Existing work often focuses on single word metaphors or lacks the nuanced human judgments needed to train robust models. This paper aims to bridge this gap by creating a new dataset and developing a deep learning model capable of predicting human judgments on metaphor paraphrases.

Methods

The authors propose a deep neural network (DNN) architecture for metaphor paraphrase classification and ranking. The core of their model consists of three main components:

  1. Two Encoders: Each encoder processes one sentence (either the metaphor or its candidate paraphrase) independently. These encoders are built using a combination of "Atrous" Convolutional Neural Networks (CNNs) and Long Short-Term Memory Recurrent Neural Networks (LSTM RNNs), followed by fully connected layers. This architecture is designed to learn compact, discriminative representations of sentences, capturing both spectral variance (CNNs) and sequential structure (LSTMs).
  2. Unified Layer: The outputs from the two encoders are merged, typically through simple concatenation, to create a single vector representing the pair of sentences.
  3. Final Layers: A set of fully connected layers operates on this merged representation to generate a judgment. For binary classification, a sigmoid function is used. For ranking, the sigmoid function's output provides a gradient that can be used to order paraphrases.

The model is trained using pre-trained Word2Vec embeddings.

Datasets

The paper introduces a newly created annotated corpus for metaphor paraphrase evaluation. This corpus consists of 200 sets, with each set containing:

  • One reference metaphorical sentence.
  • Four candidate literal paraphrases.

These candidate paraphrases are ranked on a 1-4 scale by 20 human annotators (Amazon Mechanical Turk workers) for robustness. The scale represents:

  • 1: Not paraphrases.
  • 2: Not paraphrases, but show semantic similarity.
  • 3: Loose paraphrases (some differences in style or content).
  • 4: Strong paraphrases.

The dataset is divided into four categories: Noun phrase metaphors, Adjective metaphors, Verb metaphors, and Multi-word metaphors.

Key Insights and Ideas

  • Paraphrase Ranking as a Useful Approach: The authors argue that treating metaphor paraphrase as an ordering task, rather than a simple binary classification, provides a more nuanced understanding of the relationship between metaphorical and literal expressions.
  • Deep Learning for Figurative Language: The success of their DNN model suggests that deep learning architectures, particularly those combining CNNs and LSTMs, can effectively learn complex semantic relationships involved in figurative language.
  • Transfer Learning Potential: The model demonstrates that a DNN trained for binary classification can be effectively repurposed for a gradient (ranking) task, showcasing a form of transfer learning.
  • Importance of Human Judgments: The high correlation between the model's predictions and the crowd-sourced human judgments underscores the value of carefully collected human annotations for training and evaluating NLP models on subjective tasks.
  • "Gray Areas" of Interpretation: The paper acknowledges that some paraphrase judgments fall into ambiguous "gray areas," where the distinction between a loose paraphrase and a semantically related non-paraphrase is not clear-cut. This highlights the inherent complexity of metaphor interpretation.

Key Results

  • Binary Classification: The model achieved an encouraging 75% accuracy on the binary classification task (distinguishing paraphrases from non-paraphrases) when trained on the full dataset.
  • Paraphrase Ranking: The model achieved significant correlations with human judgments on the gradient judgment prediction task:
  • Pearson correlation of 0.75

    • Spearman correlation of 0.68 These correlations were achieved on a balanced training and test set.
  • Baseline Comparison: The proposed DNN model significantly outperformed a baseline that used the cosine similarity of word vectors.

  • Model Variants: Experiments with different model configurations (e.g., without LSTM, without CNN) showed that the combined CNN-LSTM architecture was crucial for performance.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that their corpus is still under development and relatively small, which can lead to instability in results and potential overfitting. They plan to expand its size and variety.
  • "Gray Areas" and Errors: The model sometimes struggles with ambiguous cases and specific types of metaphors, such as those involving sentiment intensity, exaggeration, negation, or syntactic inversions.
  • No Explicit Alignment: The model does not use explicit "alignment" mechanisms between sentences, which, while offering flexibility, can also introduce noise. Future work may explore models with alignment.
  • Future Directions: The authors plan to conduct a more detailed error analysis, explore alternative neural network designs, and investigate the inverse task of detecting metaphors for literal sentences.

Summary updated

BibTeX
@inproceedings{bizzoni2018predicting,
  author = {Yuri Bizzoni and Shalom Lappin},
  title = {{Predicting Human Metaphor Paraphrase Judgments with Deep Neural Networks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure