TL;DR
This paper introduces two novel BERT-based models, ExampleBERT and DefinitionBERT, for detecting metaphorical language. By incorporating external knowledge from dictionary examples and definitions, guided by linguistic theories, these models achieve state-of-the-art performance on established metaphor detection datasets and offer improved interpretability.
What Problem Does This Paper Address?
Metaphor detection (MD) is a challenging Natural Language Processing (NLP) task that aims to identify metaphorical expressions in text. Understanding metaphors is crucial for deeper semantic comprehension, as it involves using knowledge from one domain to understand another. While deep learning models have shown promise, they often struggle with the nuanced understanding required for metaphor identification. Existing knowledge-based methods, though effective, haven't fully leveraged linguistic theories of metaphor. This paper addresses this gap by proposing methods that integrate external lexical knowledge (dictionary entries) with linguistic theories like the Metaphor Identification Procedure (MIP) and Select Preference Violation (SPV) to enhance metaphor detection accuracy and interpretability.
Methods
The authors propose two BERT-based models:
- ExampleBERT: This model leverages the idea that understanding the common contextual usage of a word can help identify metaphors. It uses BERT's pre-training objective (Masked Language Model) to learn contextual information from word examples found in dictionaries. The hypothesis is that by learning typical word usage, the model can better detect when a word is used in an atypical, metaphorical context.
- DefinitionBERT: This model is inspired by the MIP theory, which posits that a metaphor occurs when a word's literal meaning contrasts with its contextual meaning. DefinitionBERT directly incorporates the literal meanings of words by using their definitions from the Oxford Dictionary. It treats metaphor detection as a sentence-pair classification task, feeding context-definition pairs into BERT. The model aims to explicitly model the relationship between a word's contextual usage and its literal definitions.
Both models build upon the standard BERT architecture and are fine-tuned on metaphor detection datasets.
Datasets
The proposed models were evaluated on three widely used datasets for metaphor detection:
- VUA (Steen and Gerard): This is a large public dataset containing 10,567 sentences. The authors specifically used the VUA VERB version, which focuses on identifying metaphorical verbs.
- MOH-X (Mohammad et al., 2016): A smaller dataset with 647 sentences, sampled from WordNet, where typically only one verb per sentence is annotated.
- TroFi (Birke and Sarkar, 2006): This dataset consists of 3,737 sentences from the Wall Street Journal Corpus, with a single annotated target verb per sentence. It features a limited number of unique target verbs, leading to many training samples per verb.
Key Insights and Ideas
- Linguistic Theories as a Guide: The paper demonstrates that grounding NLP models in established linguistic theories (MIP and SPV) can significantly improve performance on tasks like metaphor detection.
- Dictionary Data is Powerful: Even seemingly simple external knowledge like dictionary definitions and examples can provide substantial benefits for metaphor detection, outperforming models that rely solely on pre-trained embeddings.
- Contextual vs. Literal Meaning: The core of metaphor detection lies in distinguishing between a word's typical contextual usage and its literal meaning. Both ExampleBERT and DefinitionBERT are designed to capture this distinction.
- Interpretability Matters: DefinitionBERT not only achieves high accuracy but also offers a degree of interpretability, allowing researchers to understand why a word is flagged as metaphorical by examining the similarity between its contextual and definitional meanings.
- Dataset Characteristics Influence Performance: The performance of the models can vary across datasets, with DefinitionBERT showing particular strength on datasets with more diverse linguistic contexts (like MOH-X) and suggesting that models might be more useful with smaller training corpora.
Key Results
- State-of-the-Art Performance: Both ExampleBERT and DefinitionBERT achieved state-of-the-art results on the VUA VERB and MOH-X datasets.
- Significant Improvements: Compared to previous state-of-the-art methods, the proposed models showed notable gains. For instance, on VUA VERB, their BERT-Baseline model already outperformed MUL-GCN by 2.28% in accuracy and 3.37% in F1-score. ExampleBERT and DefinitionBERT further improved upon this.
- DefinitionBERT's Strength: DefinitionBERT showed significant improvement on the MOH-X dataset and demonstrated strong performance on VUA VERB.
- Interpretability: The paper provides qualitative analysis showing how DefinitionBERT uses cosine similarity between contextual and definitional word embeddings to identify metaphorical usage, highlighting its interpretability. For example, the word "abuse" was correctly identified as metaphorical in "Her husband often abuses alcohol" by finding the smallest cosine similarity with its literal definition, and non-metaphorical in "This boss abuses his workers" by finding the largest similarity with its literal definition.
Limitations and Future Work
- TroFi Dataset Performance: While effective on VUA VERB and MOH-X, the models' performance on the TroFi dataset was not as strong compared to some other state-of-the-art results. The authors attribute this to the dataset's characteristics (fewer unique verbs, many samples per verb) where models might learn sufficient contextual information without needing external knowledge as much.
- Potential for Combination: The authors suggest that combining ExampleBERT and DefinitionBERT might yield even better results, though their experiments showed DefinitionBERT alone surpassed a combined approach. They hypothesize this could be due to ExampleBERT's pre-training objective weakening its ability to model pair-wise relationships.
- Future Directions:
- Exploring how to use external knowledge for sequential tasks (like VUA ALL POS).
- Investigating the use of resources from different dictionaries, ensuring definitions are non-metaphorical.
- Combining their models with other advanced techniques like Graph Convolutional Networks (GCN).
- Further applying DefinitionBERT to metaphor interpretation tasks like definition generation and lexical substitution.
Summary updated