TL;DR
Can an AI tell the difference between a poetic metaphor and total gibberish? This paper explores whether language models like BERT and RoBERTa can distinguish "creative" language from "nonsense" using plausibility scores and binary classification. The researchers found that these models are surprisingly adept at this task, achieving up to 88% accuracy, suggesting they have developed a statistical "intuition" for what makes sense—even when that sense is figurative.
What Problem Does This Paper Address?
In the world of Natural Language Understanding (NLU), metaphors represent a "cusp of comprehensibility." A metaphor like "the ATM coughed up my card" is literally false but semantically meaningful. Pure nonsense, on the other hand, is both literally false and semantically empty.
Distinguishing between the two is a high-level cognitive skill. Some scholars, like Professor Irving Massey, have even suggested that the ability to experience and distinguish metaphor from nonsense could be a "new Turing test," as it requires an aesthetic and human-centric grasp of language. This paper investigates whether modern LLMs—which are trained on massive amounts of human text—can bridge this gap or if they are simply baffled by anything that isn't literal.
Methods
The authors employed three distinct approaches to test the "intuition" of models like BERT and RoBERTa:
- Pseudo-Log-Likelihood (PLL) Scores: Since bidirectional models like BERT can't calculate traditional sentence probability (like GPT can), the authors used PLL. This involves masking one word at a time, calculating its probability based on the surrounding context, and summing these scores for the whole sentence. This acts as a proxy for how "plausible" the model finds the sentence.
- Mean Ranks of Masked Words: The researchers masked the "target" word (the metaphorical or nonsensical one) and checked where that word ranked in the model's list of predictions. If the model is "surprised" by a word, that word will have a high (worse) rank.
- Binary Classification: They fine-tuned a RoBERTa-base model to specifically categorize sentences as either "metaphorical," "literal," or "nonsense." They also compared this to a simple Naive Bayes classifier to see if the task required complex contextual understanding or just simple word-frequency patterns.
Datasets
The study utilized a combination of existing and custom-generated datasets:
- Pedinotti et al. (2021): 300 sentences split between metaphors (conventional and creative), literal sentences, and nonsense.
- Cardillo et al. (2010, 2016): 400 pairs of matched literal and metaphorical sentences.
- O’Neill et al. (2020): A corpus of sentences "without semantic context" used as a baseline for nonsense.
- Custom Nonsense: To ensure the models weren't "cheating" by spotting bad grammar, the authors generated 200 nonsense sentences by shuffling subjects/complements of metaphors or using back-translation (translating a sentence into another language and back until it loses its meaning).
Key Insights and Ideas
- The Plausibility Gradient: The models consistently ranked sentences in a specific order of "likelihood": Literal > Metaphor > Nonsense. This shows that models recognize metaphors as "less likely" than literal speech, but still significantly "more likely" than random nonsense.
- Predictability of Creativity: Even for "creative" or novel metaphors, the target words were much more predictable to the model than the words used in nonsensical sentences. This suggests that metaphors follow certain latent linguistic patterns that nonsense does not.
- Beyond Bag-of-Words: The Naive Bayes classifier (which just looks at word counts) performed poorly (22.5% accuracy), while RoBERTa excelled. This confirms that distinguishing metaphor from nonsense requires contextual awareness, not just looking for "weird" words.
Key Results
- Plausibility (PLL): In the Pedinotti dataset, literal sentences scored an average of -17.8, metaphors -26.4, and nonsense -33.1. The lower (more negative) the score, the less plausible the model finds the text.
- Predictability: The median rank for nonsensical words was vastly higher (meaning they were much harder to predict) than for metaphorical words.
- Classification Accuracy: The fine-tuned RoBERTa model achieved:
- 92.5% accuracy in distinguishing literal vs. nonsense.
- 85.0% accuracy for literal vs. metaphor.
- 75.0% to 88.0% accuracy for metaphor vs. nonsense (depending on the dataset size).
Limitations and Future Work
The authors are careful to note that while the results are impressive, they don't necessarily prove the model "understands" the metaphor in a human sense.
- Annotation Artifacts: There is a risk that the models are picking up on subtle "clues" or biases in how the datasets were constructed rather than truly evaluating semantic acceptability.
- The "Target Word" Test: Future work could involve removing the metaphorical word entirely and seeing if the model can still identify the sentence's category based on the remaining structure.
- Dataset Size: The experiments were conducted on relatively small datasets (hundreds of sentences). Testing on a larger, more diverse scale is necessary to confirm these "intuitions" hold up across all types of creative writing.
Summary updated