Distributional semantics traditionally maps words to static vectors in a high-dimensional space. While this approach captures lexical similarity—placing cat near dog—it struggles to model the compositionality of natural language. If we combine words using simple vector arithmetic like addition or multiplication, as evaluated by Diana McCarthy et al. in An Empirical Study on Compositionality in Compound Nouns, we treat words symmetrically.
However, language is inherently asymmetric and functional: modifiers actively reshape the meaning of their heads. For example, a small elephant is still massive, while a large mouse is tiny. The meaning of the adjective small cannot be a static point; it acts as a function whose output depends entirely on the noun it modifies.
To capture this interaction, a powerful paradigm treats nouns as vectors (the arguments) and functional words as linear transformations, represented as matrices. Composition is thus achieved through matrix-vector multiplication.
The Mechanics of Linear Transformations
By representing modifiers as matrices, composition shifts from simple feature aggregation to active space transformation. The core equation is straightforward:
$$\vec{v}_{\text{phrase}} = M_{\text{modifier}} \times \vec{v}_{\text{argument}}$$
Marco Baroni et al. introduced this framework in Nouns are Vectors, Adjectives are Matrices: Representing Adjective-Noun Constructions in Semantic Space, demonstrating that treating adjectives as matrices significantly improves phrase-level accuracy. This formalization of function application within vector spaces was later generalized into computational toolkits like DISSECT by Georgiana Dinu et al. in DISSECT - DIStributional SEmantics Composition Toolkit and evaluated broadly across lexical categories in General estimation and evaluation of compositional distributional semantic models.
Why does this linear mapping work so well? A matrix can scale, rotate, or project specific dimensions of a vector space. This allows it to capture complex, non-local semantic shifts. For instance, E. Dario Gutiérrez et al. demonstrated in Literal and Metaphorical Senses in Compositional Distributional Semantic Models that metaphorical adjectives (like brilliant in brilliant idea) act as systematic linear operators that shift a noun's meaning from a literal domain into a figurative one.
Furthermore, language is recursive, meaning modifiers can stack. When multiple adjectives modify a noun (e.g., rapid social change), the order of application matters. Eva Maria Vecchi et al. showed in Studying the Recursive Behaviour of Adjectival Modification with Compositional Distributional Semantics that the geometric transformation induced by an adjective matrix directly correlates with its linguistic flexibility and ordering constraints.
Beyond Adjectives: Verbs, Determiners, and Morphology
The matrix-vector framework naturally extends to other linguistic structures where words act as functions over arguments.
Logical Scope and Predicates
Grammatical words like determiners (some, no, three) do not add descriptive content but instead alter the logical scope of a concept. Raffaella Bernardi et al. showed in A relatedness benchmark to test the role of determiners in compositional distributional semantics that treating determiners as matrices transforming noun vectors captures this logical modulation far better than simple arithmetic.
For predicates like verbs, which take multiple arguments (such as a subject and an object), composition requires handling multiple inputs without causing the parameter space to explode. The authors of A practical and linguistically-motivated approach to compositional distributional semantics and Lexical Substitution for Evaluating Compositional Distributional Models designed models where verbs act as linear transformations on their arguments, allowing the predicate to systematically alter the argument's meaning while maintaining a fixed-dimensional output space.
Sub-Word Morphology
This functional perspective even applies inside words. In derivational morphology, prefixes and suffixes act as operators on a base stem (e.g., predict $\rightarrow$ predictability). Angeliki Lazaridou et al. in Compositional-ly Derived Representations of Morphologically Complex Words in Distributional Semantics and Max Kisselew et al. in Obtaining a Better Understanding of Distributional Models of German Derivational Morphology demonstrated that modeling affixes as matrices allows systems to accurately compute the meaning of rare or out-of-vocabulary derived words.
While additive models pull the representation too close to the base stem, matrix transformations successfully isolate the abstract semantic shift introduced by the affix. However, as noted by Jan Šnajder et al. in Predictability of Distributional Semantics in Derivational Word Formation, the predictability of these derived vectors remains bounded by the semantic regularity and argument structure changes of the specific morphological pattern.
Dual Representations and Deep Recurrent Networks
In full sentences, a word can simultaneously serve as an argument and an operator. To handle this dual role, Brody Huval proposed the Matrix-Vector Recursive Neural Network (MV-RNN) in Semantic Compositionality through Recursive Matrix-Vector Spaces. In this architecture, every lexical unit possesses a dual representation: a static vector capturing its intrinsic meaning, and a transformation matrix dictating how it modifies its neighbors. This allows the network to recursively evaluate complex linguistic structures, such as negation and intensification, across a syntactic tree.
This operator-based approach is highly effective for sentiment analysis. As shown by Ainur Yessenalina et al. in Compositional Matrix-Space Models for Sentiment Analysis, representing words as matrices allows iterated matrix multiplication to naturally model how intensifiers scale polarity and how negators flip sentiment values along an ordinal scale.
Iterated Matrix Composition (e.g., Sentiment Intensification):
[Not] (Matrix) × [Very] (Matrix) × [Good] (Vector) ➔ Final Phrase Vector
Remarkably, this explicit matrix-vector compositionality is implicitly mirrored in modern deep learning architectures. Xiaobing Sun 0002 et al. proved mathematically in Implicit n-grams Induced by Recurrence that the hidden states of recurrent neural networks (RNNs) function by linearizing recurrence relations into iterated matrix-vector multiplications. This mechanism allows RNNs to implicitly encode $n$-gram features and capture nuanced semantic shifts like negation without needing explicit syntactic parse trees.
Grounding and Alternative Geometries
The success of matrix-based composition relies heavily on how these transformations are learned and structured. Masayasu Muraoka et al. evaluated various frameworks in Finding The Best Model Among Representative Compositional Models and demonstrated that utilizing relation-specific transformation matrices (the Relfunc model) provides superior performance by explicitly accounting for syntactic roles.
Rather than relying solely on text corpora, these compositional functions can be grounded in real-world human-machine interactions. Federico Bianchi et al. demonstrated in Language in a (Search) Box: Grounding Language Learning in Real-World Human-Machine Interaction that learning transformation matrices from e-commerce search logs allows models to generalize effectively to unseen phrase combinations by navigating a grounded entity space.
When analyzed using human neuroimaging data, parametric semantic composition via matrices has been shown to preserve fine-grained sensory-motor attributes during phrase construction, even when abstract concept representations degrade (Shaonan Wang et al. in Investigating Inner Properties of Multimodal Representation and Semantic Compositionality With Brain-Based Componential Semantics).
Finally, alternative formulations explore relaxing the linearity constraint. For example, Kumiko Tanaka-Ishii et al. proposed in FIRE: Semantic Field of Words Represented as Non-Linear Functions representing words as non-linear functions within a continuous semantic field, showing that compositionality can still leverage additive properties while elegantly resolving word polysemy.
Key Foundational Work
Nouns are Vectors, Adjectives are Matrices: Representing Adjective-Noun Constructions in Semantic Space (2010) — contributed 2 concept elements to this direction.
About the Vectorial Calculus of the Relation Between the Semantic and Syntactic Markers of the Lexical Entry (1969) — contributed 1 concept elements to this direction.