Abstract
Distinguishing between the literal and figurative meanings of idioms—such as "break the ice"—remains a significant challenge in Natural Language Processing. This study demonstrates that the effectiveness of context representations varies significantly across different idioms. To address this, we propose a supervised late fusion ensemble method that adaptively weights lexical, topical, and distributional features. Our approach achieves state-of-the-art accuracy and improved stability across a diverse set of idiomatic expressions, outperforming single-representation models and previous state-of-the-art methods.
The Challenge of Idiomatic Context
Idioms are linguistic chameleons whose meanings shift based on context. For instance, "playing with fire" can literally refer to a child with matches or figuratively describe a risky political strategy. While humans navigate these nuances effortlessly, machines struggle due to two primary factors:
- Context Diversity: Some idioms, like "break the ice," typically appear in specific social or political contexts, while others, like "under the microscope," are found across a vast array of topics.
- Semantic Analyzability: Some idioms are semantically transparent ("in the fast lane"), where the parts hint at the whole, while others are entirely opaque ("break a leg").
Existing NLP models often rely on a single context representation—such as specific keywords or broad themes—which fails to generalize across the diverse behaviors of different idioms. As shown in Table 1, previous models often exhibit inconsistent performance across different expressions, with some models scoring as low as 0.46 on challenging idioms like "under the microscope."
Methodology: Late Fusion Ensemble
The core of our approach is a Late Fusion Ensemble Model. Instead of a "one-size-fits-all" representation, the model integrates three distinct classifiers based on different ways of "seeing" context:
- Lexical Representation: Focuses on specific surface words (Bag-of-Words) to identify nearby nouns or prepositions that signal literal usage.
- Topical Representation: Employs Latent Dirichlet Allocation (LDA) to model the broad thematic domain (e.g., sports, politics) associated with an idiom.
- Distributional Semantic Representation: Utilizes
doc2vecto capture underlying semantic meanings, allowing the model to recognize synonyms that may not match training data exactly.
The "Secret Sauce": The model uses a variant of the averaged perceptron to learn weights for these three "voters." Crucially, it incorporates a confidence measure based on the ratio of similarity between figurative and literal example sets. This allows the ensemble to dynamically shift weight toward the most certain representation for a given idiom.
Experimental Setup
We evaluated our model using the SemEval 2013 Task 5B corpus, which includes 10 target idioms (e.g., at the end of the day, break a leg, through the roof). The dataset consists of 2,371 total instances, nearly equally balanced between literal (1,185) and figurative (1,186) usages.
Key Insights
Our analysis revealed critical insights into the relationship between idiom characteristics and model performance:
- Context Diversity is a Performance Killer: We found a strong negative correlation (-0.86) between an idiom's topic diversity and the performance of lexical models. If an idiom can appear in any context, simple keyword matching fails.
- Analyzability Confuses Distributional Models: If the literal and figurative meanings are semantically similar (high analyzability), distributional models like
doc2vecstruggle to distinguish them as their vector representations become nearly identical. - Adaptability Wins: The ensemble model successfully adapts its weighting. For "break a leg" (high diversity, low analyzability), it prioritizes distributional features. For "bread and butter" (low diversity), it relies more on lexical and topical features.
Results
The Late Fusion model demonstrated superior performance compared to individual components and previous state-of-the-art methods.
Key findings include:
- Higher F1 Scores: The model achieved figurative usage F1 scores between 0.68 and 0.85, consistently outperforming individual models.
- Stability: Unlike previous models that showed erratic performance, our ensemble maintained a high and narrow performance range across all tested idioms.
- Statistical Significance: Improvements over the "Best Other" models were statistically significant for 8 out of the 10 idioms tested.
Conclusion
This work highlights the necessity of adaptive context representations for idiom recognition. By fusing lexical, topical, and distributional information, our model provides a robust solution to the variability of idiomatic language. Future work will explore more complex linguistic features, such as syntactic dependencies, and evaluate the model's scalability to a broader range of idioms beyond the SemEval dataset.
Summary updated