Introduction
Large language models (LLMs) have demonstrated remarkable capabilities in capturing semantic similarity and performing complex linguistic tasks. However, their ability to handle compositionality—the principle that the meaning of a complex expression is determined by the meanings of its constituent parts and the rules used to combine them—remains a significant "black box." This paper investigates the "modifier phenomena" in formal linguistics, specifically how different types of adjectives modify nouns. For example, an intersective adjective like "red" implies a "red car" is both red and a car, whereas a privative adjective like "fake" implies a "fake wall" is specifically not a wall. We introduce a framework to measure whether neural embeddings respect these logical boundaries by translating Montague semantics into geometric constraints.
Methodology
The core of our approach is the "Embedding-Denotation Analogy." We hypothesize that distances in a model's vector space should reflect relationships in formal set theory. This bridge between formal logic and geometry is established using metamorphic relations.
We propose three primary tests for consistency:
- Single Phrase Intersectivity (SPI): If an adjective is intersective, the embedding of the phrase (e.g., "red car") should be closer to its components ("red" and "car") than the components are to each other.
- Phrase Pair Intersectivity (PPI): This tests if phrases sharing an intersective adjective (e.g., "Canadian writer" and "Canadian surgeon") are closer together than phrases sharing a non-intersective one (e.g., "skillful writer" and "skillful surgeon"). This test is more robust against simple mathematical artifacts.
- Non-subsectivity Test: This measures whether the adjective "pulls" the phrase embedding closer to itself than to the noun. For non-subsective adjectives like "fake," the phrase should logically move further away from the original noun's meaning.
Our benchmark dataset consists of 44,652 adjective-noun (AN) and adjective-adjective-noun (AAN) phrases. We utilized 122 adjectives categorized into Intersective, Subsective, Privative, Plain Non-Subsective, and Ambiguous, along with 12 nouns covering both concrete and abstract concepts.
Results and Analysis
Our evaluation of models ranging from classic static embeddings (Word2Vec, GloVe) to state-of-the-art (SOTA) transformers like Stella and NV-Embed-v2 revealed several critical insights. While many modern models achieve high scores on basic intersectivity tests, these results are often confounded by "mean-pooling bias." This is a mathematical artifact where the phrase vector behaves like a simple average of its components, mimicking intersectivity without true semantic understanding.
As shown in the table above, while some models like Stella and TE3-small show perfect scores in certain categories, others like DPR and Specter show significant variance. Furthermore, larger models tend to exhibit "Adjective Dominance," where the modifier's vector significantly outweighs the noun's. While this helps them pass some subsectivity tests, it suggests they might be "over-focusing" on modifiers at the expense of the core concept. Most models showed high consistency scores (0.85–1.0) for single phrases, but this dropped significantly in more complex phrase-pair tests, where scores for SOTA models hovered around 0.50–0.91 depending on the adjective category.
Conclusion
This study highlights a fundamental gap between neural model behavior and formal linguistic theory. No model tested consistently adhered to Montagovian logic across all categories. This suggests that neural models may capture a different kind of "distributional logic" or simply lack the capability to represent fine-grained linguistic distinctions in limited contexts. Limitations include the focus on short phrases and the reliance on cosine distance as a proxy for set inclusion. Future work should investigate these phenomena in larger contexts and explore alternative geometric representations, such as cone-based embeddings, to better capture the nuances of linguistic compositionality.
Summary updated