TL;DR
This paper investigates how well pre-trained language models (PLMs) understand figurative language by analyzing their learning trajectories. They found that models learn to recognize hyperboles and oxymorons relatively early in training, while metaphors and especially pleonasms remain challenging. Interestingly, larger models generally perform better, and the specific layers where this knowledge is stored are often in the middle to later stages of the model.
What Problem Does This Paper Address?
Understanding figurative language, like metaphors and hyperboles, is a cornerstone of human communication, requiring deep semantic understanding and real-world knowledge. While PLMs have shown remarkable capabilities in various NLP tasks, it's still debated whether they truly "understand" language or just master statistical patterns. Figurative language, with its nuanced and non-literal usage, presents a particularly difficult challenge for evaluating this understanding. This paper aims to shed light on how PLMs acquire and represent knowledge about specific figures of speech, moving beyond just assessing their final performance to understanding their learning process.
Methods
The core of this research uses probing tasks to assess PLMs' understanding of figurative language. A probe is essentially a simple classifier trained on the embeddings (numerical representations) generated by a PLM. If the classifier can accurately predict whether a sentence contains a specific figure of speech, it suggests that this information is encoded in the PLM's embeddings.
To address the limitations of traditional accuracy-based probing (which can be misleading), the authors employ the Minimum Description Length (MDL) probing method. MDL is an information-theoretic approach that measures not only the classifier's performance but also the "effort" required to achieve it, making it more robust to random chance and probe settings.
Crucially, the study analyzes learning trajectories by evaluating the PLMs at various checkpoints throughout their training process, not just on the final, fully trained models. This allows them to track when and how figurative language understanding emerges. They also extend this analysis to individual layers within the PLMs to pinpoint where this knowledge is stored.
Datasets
The study utilizes existing datasets for four figures of speech:
- Hyperbole: A dataset of 1396 sentences, balanced between hyperbolic and literal examples, adapted from Troiano et al. (2018).
- Metaphor: A binary version of the Language Computer Corporation (LCC) dataset, with 1600 sentences.
- Oxymoron: A newly created dataset of 1564 sentences, including positive instances from Xu et al. (2023) and synthetically generated negative instances using GPT-4.
- Pleonasm: A dataset of 3002 sentences derived from the Semantic Pleonasm Corpus (SPC) by Kashefi et al. (2018), adapted for binary classification.
For training models from scratch, they use:
- Project Gutenberg: A large corpus of books (approx. 13GB) hypothesized to contain more literary language.
- Wikipedia: A generic corpus (approx. 11GB).
- The Pile: A large, diverse dataset (approx. 800GB) used for pre-training the main GPT-NeoX models.
Key Insights and Ideas
- Early Learning for Some, Persistent Struggle for Others: Models learn to recognize hyperboles and oxymorons relatively early in training, with performance stabilizing quickly. Metaphors are more challenging, and pleonasms remain particularly difficult for the models tested.
- The "Middle" is Where the Magic Happens (Sometimes): For figurative language, the best-performing layers are often found in the central or second half of the model, rather than exclusively the final layer. This suggests that while the final layer might focus on the core language modeling task, semantic nuances like figurative language are distributed across intermediate layers.
- Bigger is (Usually) Better: Larger models generally achieve higher compression scores, indicating a better grasp of figurative language. However, there are exceptions, and the learning curves can sometimes plateau or even slightly degrade for very large models.
- Literary Language Isn't Necessarily Better: Training on a literary corpus like Project Gutenberg did not confer a significant advantage in learning figurative language compared to a more generic Wikipedia corpus. This suggests that the sheer volume and diversity of data (like The Pile) might be more critical than its specific literary quality for this task.
- Oxymoron's Peculiar Trajectory: For oxymorons, while the final layer's performance can degrade in later training stages (a phenomenon akin to forgetting), the intermediate layers seem to retain and even improve this knowledge, suggesting it's not catastrophic forgetting but rather a shift in the final layer's focus.
Key Results
- Hyperbole and Oxymoron are Most Recognizable: The models achieved the highest compression scores for hyperbole and oxymoron, with some models reaching compression values above 4.5 for oxymoron.
- Metaphor and Pleonasm are Challenging: Metaphor recognition yielded lower compression scores (around 1.6 for the largest models), and pleonasm recognition showed little improvement over random initialization.
- Learning Stabilizes Early: For hyperbole, oxymoron, and metaphor, most models showed significant compression gains within the first 10 billion tokens of training, with minimal improvements thereafter.
- Layer Performance Varies: Table 1 shows that for models like 1.4B and 6.9B, the best layers for hyperbole were around layer 13 and 18, respectively, significantly outperforming the last layer. For oxymoron, the 160M model showed a remarkable 206.8% improvement in compression when considering its best layer compared to the last.
- Dataset Impact: Models trained on the larger "The Pile" dataset consistently outperformed those trained on smaller Project Gutenberg or Wikipedia datasets, especially for hyperbole and oxymoron.
Limitations and Future Work
The authors acknowledge several limitations:
- Model Accessibility: Their method relies on accessing model embeddings and checkpoints, making it inapplicable to closed-source LLMs.
- Probing Task Limitations: Probing tasks reveal what knowledge is encoded but not necessarily how it's used in text generation.
- Dataset Size and Diversity: The study used only one dataset per figure of speech, and all data was in English. Future work could explore multiple datasets of varying sizes and languages.
- Model Family: The analysis was primarily conducted on GPT-NeoX models. Expanding to other LLM families would be beneficial.
Future work could involve:
- Expanding the analysis to more figures of speech, datasets, and languages.
- Investigating multilingual models to see if figurative language knowledge is language-specific or shared.
- Exploring LLMs beyond the GPT-NeoX family.
- Further investigating the phenomenon of "forgetting" in the last layer versus knowledge retention in intermediate layers for specific figures of speech.
Summary updated