Most natural language processing systems operate on a fundamental premise: the principle of compositionality. This principle states that the meaning of a complex expression is a direct function of the meanings of its parts and the syntactic rules used to combine them. From early phrasal systems like the one introduced by Wilensky et al. in PHRAN - A Knowledge-Based Natural Language Understander to modern Transformer-based large language models, NLP architectures rely heavily on this bottom-up aggregation. Whether summing word vectors or computing multi-head self-attention over sub-word tokens, models assume that text can be understood by systematically combining smaller semantic units.
Where Compositionality Fails
Human language is filled with constructs that violate this assumption: non-compositional expressions. These include idioms ("kick the bucket," "cut corners"), metaphors, and certain multi-word expressions (MWEs) or compounds ("baby oil," "hot dog"). As Bhat et al. demonstrate in PIE: A Parallel Idiomatic Expression Corpus for Idiomatic Sentence Generation and Paraphrasing, the holistic meaning of a highly non-compositional idiom bears virtually no relation to its individual constituent words.
Why does this break modern architectures? Consider how a standard Transformer processes a phrase. Its attention mechanism computes contextualized representations by looking at individual tokens. For an arbitrary, culturally dependent idiom, there is no statistical or logical path from the token "bucket" to the concept of "death." As Stowe et al. observe in IMPLI: Investigating NLI Models' Performance on Figurative Language, while metaphors often retain some compositional logic, idioms represent an entirely arbitrary mapping. Purely additive or aggregative models, even when augmented with noise terms as explored by Cotterell et al. in Morphological Smoothing and Extrapolation of Word Embeddings, fundamentally struggle to map these holistic meanings accurately.
The Cost of Literalism
When a model encounters a non-compositional phrase without explicit figurative awareness, it defaults to a literal interpretation. In A Long Hard Look at MWEs in the Age of Language Models, Vered Shwartz points out that this literalism creates factually incorrect internal representations, leading to severe failures in downstream reasoning.
These failures cascade across various NLP applications:
- Machine Translation: Word-for-word substitutions yield nonsensical outputs, a problem highlighted by I. Dan Melamed in Automatic Discovery of Non-Compositional Compounds in Parallel Data and Saxena et al. in EPIE Dataset: A Corpus For Possible Idiomatic Expressions.
- Abstractive Summarization: Models frequently misinterpret figurative language, producing distorted summaries (Chopra et al. in Abstractive Sentence Summarization with Attentive Recurrent Neural Networks).
- Writing Assistance: Even state-of-the-art models providing writing assistance in complex, culturally rich languages like Arabic struggle to move past literal definitions, missing crucial contextual nuances (Magdy et al. in Gazelle: An Instruction Dataset for Arabic Writing Assistance).
- Sentiment Analysis: As Jia et al. note in Hierarchical Gated Deep Memory Network With Position-Aware for Aspect-Based Sentiment Analysis, word-level processing models can easily misclassify the sentiment of a sentence if an idiom contains individual words with misleading polarities.
Disambiguation and the Spectrum of Meaning
Compounding the challenge is the fact that many multi-word expressions are semantically ambiguous; they can be interpreted either literally or idiomatically depending on the context. The authors of Japanese Idiom Recognition: Drawing a Line between Literal and Idiomatic Meanings address this boundary, emphasizing the need for models to dynamically distinguish between these senses.
In fact, multi-word expressions exist on a continuous spectrum. In Verb Noun Construction MWE Token Classification, Mona T. Diab explains that MWEs range from completely transparent and compositional to entirely opaque and non-compositional. Interestingly, supervised models often find the literal usages of potentially idiomatic phrases harder to identify than their idiomatic counterparts, suggesting that literal expressions exhibit higher linguistic variability or require more sophisticated contextual features to isolate.
Strategies for Non-Compositional Awareness
To bridge this gap, researchers have developed several distinct strategies to make models "idiom-aware."
Dual Embeddings and Hybrid Representations
One approach is to explicitly model both compositional and non-compositional pathways. The authors of Adaptive Joint Learning of Compositional and Non-Compositional Phrase Embeddings developed a framework that adaptively blends representations by learning a "compositionality score" to decide whether a phrase should be treated as a single holistic unit or parsed word-by-word.
Similarly, for Chinese idiom (Chengyu) prediction, Jiang et al. utilize a dual embedding model in A BERT-based Dual Embedding Model for Chinese Idiom Prediction that splits representations into local grammatical features and global semantic meanings. Treating idioms as unique, holistic semantic units within a context allows models to vastly outperform standard word aggregations, as shown by Jiang in Learning and Evaluating Chinese Idiom Embeddings.
Contrastive Learning and Structural Separation
Rather than just blending vectors, some frameworks actively force the model to separate literal and figurative spaces. In CLCL: Non-compositional Expression Detection with Contrastive Learning and Curriculum Learning, Zhou et al. employ contrastive learning to pull literal and figurative representations apart in the embedding space, training on a curriculum of increasingly difficult examples to improve generalization to unseen expressions.
External Knowledge Graphs
When statistical patterns in text are insufficient, external commonsense knowledge can provide the missing link. Zeng et al. introduced IEKG: A Commonsense Knowledge Graph for Idiomatic Expressions, which maps explicit "if-then" reasoning to figurative meanings. This allows models to look up the abstract implications of an idiom (e.g., mapping "licking someone's boots" to "insincerity") rather than relying on standard compositional semantics.
Pre-processing and Paraphrasing
Another pragmatic direction involves "de-idiomatizing" text before it reaches downstream tasks. Sneha Mehta demonstrates in Simplify-Then-Translate: Automatic Preprocessing for Black-Box Translation that translating complex, non-compositional phrases into literal equivalents via an automatic preprocessing step significantly boosts the accuracy of black-box translation systems. This is supported by Zhou et al. in Idiomatic Expression Paraphrasing without Strong Supervision, who leverage dictionary definitions and iterative back-translation to substitute figurative expressions with literal paraphrases without requiring massive parallel corpora.