Introduction
Figurative language—encompassing sarcasm, metaphors, idioms, and similes—presents a significant challenge for natural language processing (NLP). Unlike literal language, the intended meaning of figurative speech often diverges from or directly opposes the literal words used. Humans navigate this "nightmare" of ambiguity through pragmatic enrichment, using contextual cues, sentiment, and semantic incongruity to infer intent.
While modern Transformer-based Language Models (TLMs) achieve impressive performance on these tasks, they are often criticized as "black boxes." It remains unclear whether these models are learning genuine linguistic nuances or merely exploiting statistical shortcuts. This study bridges psycholinguistics and machine learning to evaluate whether AI models and humans utilize similar linguistic strategies.
Methodology
The researchers framed the problem as a four-way classification task involving sarcasm, simile, idiom, and metaphor. They compared two distinct groups of models:
- Black-Box Models: Fine-tuned versions of BERT, RoBERTa, and XLNet.
- White-Box Models: Interpretable baselines, including Logistic Regression (LR) and Random Forest (RF) using Tf-idf vectors.
To analyze the decision-making process of the black-box models, the team employed SHAP (Shapley Additive Explanations) to identify influential features. These features were then mapped to linguistic categories using the LIWC (Linguistic Inquiry and Word Count) framework. Finally, an online experiment with 15 native English speakers was conducted to compare human feature selection with model behavior.
Dataset
The study utilized the FLUTE (Figurative Language Understanding through Textual Explanations) dataset. The data was reorganized to focus exclusively on figurative sentences, removing literal premises to ensure models were identifying figurative characteristics rather than performing simple inference. The final set included 2,212 instances of sarcasm, 625 similes, 884 idioms, and 621 metaphors.
Linguistic Feature Analysis
A key finding of the study is the varying alignment between models and human linguistic intuition. While BERT and XLNet often mirror human behavior—particularly for "apparent cue" classes like similes—other models show surprising divergence.
Key Insights:
- The RoBERTa Mystery: Despite high accuracy, RoBERTa's strategy is non-intuitive. While BERT and XLNet correctly identify "like/as" for similes, RoBERTa focuses on seemingly random tokens, suggesting that high performance does not always equate to interpretability.
- Verbs and Metaphors: All Transformer models showed a strong preference for verbs when identifying metaphors. This aligns with the psycholinguistic theory of "predicative metaphors," where the verb serves as the primary vehicle for non-literal meaning (e.g., "The economy is limping").
- Filtering Noise: Black-box models excel at ignoring high-frequency function words (like the or of) to focus on salient content words, whereas white-box models are often distracted by these common tokens.
Performance Evaluation
In terms of raw performance, black-box models significantly outperformed white-box baselines. BERT and RoBERTa both achieved a 0.95 Macro-F1 score, compared to 0.87 for the best-performing Logistic Regression model.
Both humans and models found idioms and metaphors more "opaque" (difficult to classify) than sarcasm and similes. Sarcasm and similes benefit from explicit markers—such as sentiment words or comparison terms like "like"—making them easier to detect. In contrast, idioms and metaphors often require broader contextual or cultural knowledge that a single sentence may not provide.
Conclusion
This research highlights that while Transformer models can achieve near-human performance in figurative language classification, their underlying strategies vary. BERT and XLNet show promising alignment with human linguistic cues, whereas RoBERTa achieves similar results through less interpretable means. Future work should explore non-formulaic sarcasm and the role of broader context in mastering opaque linguistic categories like idioms.
Summary updated