Distributional semantics allows us to represent the meaning of a word as a point in a high-dimensional vector space. While these "word embeddings" capture lexical similarity—knowing that coffee is closer to tea than to bicycle—they struggle to represent the meaning of phrases. Language is inherently compositional: the meaning of "hot pepper" is more than just the sum of its parts.
To solve this, researchers have developed Compositional Distributional Semantic Models (CDSMs). These models use algebraic operations to combine individual word vectors into complex representations for phrases and sentences.
Addition as Semantic Accumulation
The simplest way to combine two vectors is to add them. In an additive model, the meaning of a phrase is a weighted sum of its constituents. Jeff Mitchell and Mirella Lapata demonstrated in Syntactic and Semantic Factors in Processing Difficulty: An Integrated Measure that this approach is surprisingly effective for capturing the aggregate context of a sentence fragment.
The intuition behind addition is accumulation. If the vector for bank contains features for both "finance" and "river," adding the vector for water reinforces the "river" dimensions while the "finance" dimensions remain static. Diana McCarthy and Suresh Manandhar found that additive models consistently outperform multiplicative ones when modeling the "literality" of compound nouns in An Empirical Study on Compositionality in Compound Nouns, as they preserve the core semantic features of both words.
Multiplication as Feature Filtering
While addition accumulates meaning, element-wise multiplication (the Hadamard product) acts as a logical filter. If a specific dimension has a high value in both words, it is amplified; if it is near zero in either, it is suppressed.
In Using landing pages for sponsored search ad selection, Yejin Choi and colleagues used multiplication to bridge the gap between user queries and ads. Multiplication acts as a "semantic intersection," ensuring that the resulting vector only retains features common to all terms. This prevents a single dominant word (like a brand name) from skewing the entire representation. Similarly, Jun-Ping Ng used multiplicative composition in Better Summarization Evaluation with Word Embeddings for ROUGE to compare the semantic content of n-grams, allowing for more flexible evaluation than rigid string matching.
Words as Functional Operators
Simple arithmetic treats all words as equal, but language is often asymmetric. An adjective like red modifies a noun like car, but the reverse is rarely true. To capture this, researchers like Richard Socher and Christopher D Manning proposed representing words as both vectors and transformation matrices.
In Semantic Compositionality through Recursive Matrix-Vector Spaces, they introduced the MV-RNN, where each word acts as a linear operator on its neighbor. This allows for complex semantic shifts:
- Negation: A word like not can be a matrix that flips the sentiment of the following vector.
- Intensification: A word like very can scale the magnitude of specific attributes.
This functional approach is particularly useful for morphological derivation. Jan Šnajder and Sebastian Padó explored this in Predictability of Distributional Semantics in Derivational Word Formation, showing that while addition preserves the base word's meaning, matrix-based models are better at capturing the abstract functional change introduced by a prefix or suffix (e.g., turning predict into predictability).
Grounding and the Binding Problem
The ultimate test for these models is whether they align with human cognition and the physical world. Shaonan Wang and Jiajun Zhang used fMRI data in Investigating Inner Properties of Multimodal Representation and Semantic Compositionality With Brain-Based Componential Semantics to show that matrix-based composition more accurately reflects how the human brain integrates information than simple addition.
However, even sophisticated models face the binding problem. Martha Lewis and Nihal V. Nayak found in Does CLIP Bind Concepts? Probing Compositionality in Large Image Models that modern vision-language models often fail to distinguish between "a red cube and a blue sphere" and "a blue cube and a red sphere." They argue that without explicit algebraic structures—like circular convolution or matrix multiplication—models revert to a "bag of concepts," failing to truly bind attributes to objects.
By moving from static word points to dynamic algebraic operations, compositional distributional semantics provides a framework for understanding how the infinite variety of human thought is constructed from a finite vocabulary.