On the Cusp of Comprehensibility: Can Language Models Distinguish Between Metaphors and Nonsense?

Bernadeta Griciūtė, Marc Tanti, Lucia Donatelli2022 · DOI

TL;DR

Can an AI tell the difference between a poetic metaphor and total gibberish? This paper explores whether language models like BERT and RoBERTa can distinguish "creative" language from "nonsense" using plausibility scores and binary classification. The researchers found that these models are surprisingly adept at this task, achieving up to 88% accuracy, suggesting they have developed a statistical "intuition" for what makes sense—even when that sense is figurative.

What Problem Does This Paper Address?

In the world of Natural Language Understanding (NLU), metaphors represent a "cusp of comprehensibility." A metaphor like "the ATM coughed up my card" is literally false but semantically meaningful. Pure nonsense, on the other hand, is both literally false and semantically empty.

Distinguishing between the two is a high-level cognitive skill. Some scholars, like Professor Irving Massey, have even suggested that the ability to experience and distinguish metaphor from nonsense could be a "new Turing test," as it requires an aesthetic and human-centric grasp of language. This paper investigates whether modern LLMs—which are trained on massive amounts of human text—can bridge this gap or if they are simply baffled by anything that isn't literal.

Methods

The authors employed three distinct approaches to test the "intuition" of models like BERT and RoBERTa:

  1. Pseudo-Log-Likelihood (PLL) Scores: Since bidirectional models like BERT can't calculate traditional sentence probability (like GPT can), the authors used PLL. This involves masking one word at a time, calculating its probability based on the surrounding context, and summing these scores for the whole sentence. This acts as a proxy for how "plausible" the model finds the sentence.
  2. Mean Ranks of Masked Words: The researchers masked the "target" word (the metaphorical or nonsensical one) and checked where that word ranked in the model's list of predictions. If the model is "surprised" by a word, that word will have a high (worse) rank.
  3. Binary Classification: They fine-tuned a RoBERTa-base model to specifically categorize sentences as either "metaphorical," "literal," or "nonsense." They also compared this to a simple Naive Bayes classifier to see if the task required complex contextual understanding or just simple word-frequency patterns.

Datasets

The study utilized a combination of existing and custom-generated datasets:

  • Pedinotti et al. (2021): 300 sentences split between metaphors (conventional and creative), literal sentences, and nonsense.
  • Cardillo et al. (2010, 2016): 400 pairs of matched literal and metaphorical sentences.
  • O’Neill et al. (2020): A corpus of sentences "without semantic context" used as a baseline for nonsense.
  • Custom Nonsense: To ensure the models weren't "cheating" by spotting bad grammar, the authors generated 200 nonsense sentences by shuffling subjects/complements of metaphors or using back-translation (translating a sentence into another language and back until it loses its meaning).

Key Insights and Ideas

  • The Plausibility Gradient: The models consistently ranked sentences in a specific order of "likelihood": Literal > Metaphor > Nonsense. This shows that models recognize metaphors as "less likely" than literal speech, but still significantly "more likely" than random nonsense.
  • Predictability of Creativity: Even for "creative" or novel metaphors, the target words were much more predictable to the model than the words used in nonsensical sentences. This suggests that metaphors follow certain latent linguistic patterns that nonsense does not.
  • Beyond Bag-of-Words: The Naive Bayes classifier (which just looks at word counts) performed poorly (22.5% accuracy), while RoBERTa excelled. This confirms that distinguishing metaphor from nonsense requires contextual awareness, not just looking for "weird" words.

Key Results

  • Plausibility (PLL): In the Pedinotti dataset, literal sentences scored an average of -17.8, metaphors -26.4, and nonsense -33.1. The lower (more negative) the score, the less plausible the model finds the text.
  • Predictability: The median rank for nonsensical words was vastly higher (meaning they were much harder to predict) than for metaphorical words.
  • Classification Accuracy: The fine-tuned RoBERTa model achieved:
  • 92.5% accuracy in distinguishing literal vs. nonsense.
  • 85.0% accuracy for literal vs. metaphor.
  • 75.0% to 88.0% accuracy for metaphor vs. nonsense (depending on the dataset size).

Limitations and Future Work

The authors are careful to note that while the results are impressive, they don't necessarily prove the model "understands" the metaphor in a human sense.

  • Annotation Artifacts: There is a risk that the models are picking up on subtle "clues" or biases in how the datasets were constructed rather than truly evaluating semantic acceptability.
  • The "Target Word" Test: Future work could involve removing the metaphorical word entirely and seeing if the model can still identify the sentence's category based on the remaining structure.
  • Dataset Size: The experiments were conducted on relatively small datasets (hundreds of sentences). Testing on a larger, more diverse scale is necessary to confirm these "intuitions" hold up across all types of creative writing.

Summary updated

Abstract

Utterly creative texts can sometimes be difficult to understand, balancing on the edge of comprehensibility. However, good language skills and common sense allow advanced language users both to interpret creative texts and to reject some linguistic input as nonsense. The goal of this paper is to evaluate whether the current language models are also able to make the distinction between a creative language use and nonsense. To test this, we have computed mean rank and pseudo-log-likelihood score (PLL) of metaphorical and nonsensical sentences, and fine-tuned several pretrained models (BERT, RoBERTa) for binary classification between the two categories. There was a significant difference in the mean ranks and PPL scores of the categories, and the classifier reached around 85.5% accuracy. The results raise further questions on what could have let to such satisfactory performance.
BibTeX
@misc{griciute2022cusp,
  author = {Bernadeta Griciūtė and Marc Tanti and Lucia Donatelli},
  title = {{On the Cusp of Comprehensibility: Can Language Models Distinguish Between Metaphors and Nonsense?}},
  year = {2022},
  doi = {10.18653/v1/2022.flp-1.25},
}
Figure