TL;DR
This paper investigates how BERT, a powerful language model, understands idiomatic expressions. By masking parts of sentences and observing BERT's predictions, they found that BERT primarily relies on the idiomatic expression itself but also uses surrounding context. This work helps us understand how BERT processes figurative language and distinguishes it from literal meaning.
What Problem Does This Paper Address?
Understanding and processing idioms is a long-standing challenge in Natural Language Processing (NLP). Idioms are phrases where the meaning isn't deducible from the individual words (e.g., "kick the bucket"). A key difficulty is that many expressions can be used both literally and idiomatically, making it hard for NLP systems to distinguish between these uses. This paper focuses on a specific aspect of this problem: where in a sentence does a model like BERT find the "signal" that an expression is being used idiomatically? This is analogous to the psycholinguistic concept of an "idiomatic key" in human language processing.
Methods
The researchers employed a "probing" methodology combined with input masking techniques.
- Probing: They trained a simple classifier (a Multi-Layer Perceptron or MLP) on BERT's sentence embeddings to predict whether a sentence contained an idiomatic or literal expression. If the MLP could accurately predict idiomaticity from the embeddings, it suggested BERT's embeddings encoded this information.
- Input Masking: To pinpoint where BERT found this idiomatic information, they systematically masked parts of the input sentences before generating embeddings.
- Word Masking: Replaced words with a
[MASK]token, completely removing their information. - Embedding Masking: Masked the word embeddings themselves, but still included the words in the input. This allowed BERT's self-attention mechanism to still indirectly consider the masked words.
- Differential Idiomaticity: They measured how the MLP's idiomaticity score changed when a part of the sentence was masked compared to when it was present. A larger change indicated that the masked part was more informative for BERT's understanding of idiomaticity.
Datasets
The primary dataset used was the VNIC dataset (Cook et al., 2008).
- It contains 2984 sentences with 56 idiomatic expressions.
- Sentences are labeled as idiomatic, literal, or unknown.
- To address class imbalance (more idiomatic than literal sentences), the researchers created balanced datasets by downsampling the idiomatic sentences. They generated 20 different versions of the dataset for robust evaluation and cross-validation.
Key Insights and Ideas
- BERT's Idiomatic Key is Local, but Not Exclusive: BERT primarily finds the signal for idiomatic usage within the idiomatic expression itself. However, it also draws information from the surrounding context.
- Distinguishing Disruption from Incongruity: BERT appears capable of differentiating between the disruption caused by missing words and the specific incongruity introduced by an idiomatic expression. This is a crucial step towards truly understanding figurative language.
- Word Masking vs. Embedding Masking: The fact that word masking had a larger impact than embedding masking suggests that while BERT's self-attention can infer context, the direct presence of the words within the expression is more critical for identifying idiomaticity.
- Literal Sentences are Sensitive to Context Disruption: For literal sentences, masking random words outside the expression significantly impacted the idiomaticity score, suggesting that disrupting the expected literal context can be misinterpreted by BERT.
Key Results
- The baseline MLP model achieved high performance, correctly predicting idiomatic sentences with a score of 0.85 (ideal 1.00) and literal sentences with a score of 0.17 (ideal 0.00), confirming that BERT embeddings encode idiomaticity information.
- Masking the target idiomatic expression (using word masking) resulted in a significant drop in the idiomaticity score (differential idiomaticity of 0.06), indicating its importance.
- Masking random words outside the expression also caused a statistically significant difference (differential idiomaticity of 0.02), highlighting the role of context.
- For literal sentences, masking the target expression led to a significant negative differential idiomaticity (-0.08), suggesting that disrupting the literal expression leads BERT to perceive it as less literal. Masking random words outside the expression had a negligible effect on literal sentences.
Limitations and Future Work
- Disentangling Disruption: The authors acknowledge that it's challenging to definitively disentangle the effect of "disruption" (e.g., missing words) from the specific "incongruity" of an idiom. They suggest that their findings hint at BERT's ability to do so, but further work is needed.
- Future Research Directions:
- Investigate the role of surrounding context further by masking different categories of words (e.g., content words, topical key words, words by part-of-speech).
- Explore how BERT handles other types of figurative language.
- Further probe the nature of the "idiomatic key" and how BERT's self-attention mechanism contributes to its understanding.
Summary updated