Standard distributional semantics excels at capturing the meaning of individual words by mapping them to vectors in a continuous space. However, when it comes to combining these words into phrases or sentences, traditional approaches often fall back on simple vector addition or multiplication. This treats language as a "bag of words," completely discarding syntax and structure. Under an additive model, the sentence "dog bites man" maps to the exact same representation as "man bites dog", and the subtle contextual shift between "throwing a football" and "throwing a bomb" is lost.
To bridge the gap between data-driven word vectors and the structural rules of formal logic, a powerful paradigm has emerged: tensor-based compositional semantics. Instead of treating every word uniformly as a static point in space, this framework assigns distinct mathematical types to words based on their syntactic roles. Nouns act as basic vectors, while relational words—such as adjectives, verbs, and prepositions—are modeled as higher-order tensors that function as linear or multi-linear operators.
Relational Words as Linear Maps
The core intuition behind this approach is that modifiers and predicates do not simply add to the meaning of a noun; they transform it.
This mechanism was pioneered by Baroni et al. (2010), who proposed representing nouns as vectors and adjectives as matrices (second-order tensors). When an adjective like "electronic" modifies a noun like "community", the matrix acts as a linear map that multiplies the noun vector, producing a new composite vector in the noun space.
This type-driven framework scales naturally to more complex grammatical structures. Grefenstette et al. (2011) provided the first large-scale experimental validation for this categorical model, demonstrating that treating relational words as tensors allows sentence construction to treat syntax as a direct guide for linear algebraic operations. For example, a transitive verb can be modeled as a third-order tensor. As shown by Polajnar et al. (2014), this tensor functions as a multi-linear map that transforms subject and object noun vectors into a sentence-level representation via tensor contraction:
$$\text{Sentence Vector} = \mathcal{V} \times_{\text{subject}} \vec{v}_{\text{subj}} \times_{\text{object}} \vec{v}_{\text{obj}}$$
By making the mathematical structure mirror syntactic dependencies, as validated across various neural word representations by Milajevs et al. (2014), these models preserve structural relations that holistic sentence encoders or bag-of-words baselines often obscure.
Binding Meanings to Structural Roles
To prevent structural information from washing away during composition, researchers utilize Tensor Product Representations (TPRs) to explicitly bind lexical meanings to their grammatical roles.
Instead of compressing a sentence into a single vector sum, the architecture developed by Palangi et al. (2018) decomposes language into semantic "fillers" (what a word means) and structural "roles" (its position or grammatical function). By taking the tensor product ($\otimes$) of a filler vector and a role vector, the model creates an independent, structured representation for each constituent:
$$\text{Bound Representation} = \vec{v}_{\text{filler}} \otimes \vec{v}_{\text{role}}$$
This multi-linear approach ensures that identical tokens are explicitly differentiated based on their structural positions. Crucially, this high-order representation allows for an unbinding operation. As demonstrated by Chen et al. (2020), applying an unbinding operation to high-order tensors enables the precise extraction of symbolic relations and their arguments from a continuous vector space. This mechanism provides a mathematical bridge between continuous neural representations and the discrete requirements of formal-language generation.
Multiplicative Interactions and Contextual Shifts
Why are tensor products and contractions fundamentally more expressive than vector addition? The answer lies in multiplicative interactions.
When two ambiguous or polysemous words interact, simple addition merely averages their features. In contrast, Zhang et al. (2018) note that tensor products capture the exponential variety of combined meanings, accounting for all possible combinations of their individual semantic states. This allows small changes in event arguments to significantly shift the overall context. For instance, Weber et al. (2018) showed that treating predicates as algebraic operators via tensor composition allows a verb's representation to dynamically adapt depending on whether its object is a "football" or a "bomb".
Vector Addition (Linear):
[throwing] + [football] --> Merges average features (loses context-specific shifts)
Tensor Contraction (Multiplicative):
[throwing tensor] × [football vector] --> Transforms the operator space into a specific "sports" context
This capacity to capture non-linear semantic transformations is highly effective across diverse linguistic structures:
- Tree-Structured Composition: Socher et al. (2013) introduced the Recursive Neural Tensor Network (RNTN), which uses a tensor to define multiple bilinear forms for child nodes in a parse tree. This enables the model to accurately capture complex semantic transformations, such as how a negation word changes the sentiment of an entire phrase.
- Multi-Way Grammatical Relations: Van de Cruys et al. (2013) modeled Subject-Verb-Object (SVO) triples using a multi-mode tensor where separate modes represent each grammatical role. This allows the joint interaction of subjects and objects to simultaneously constrain the meaning of the verb, rather than relying on independent pairwise operations.
- Structural Resolution: In tasks involving structural omission, such as verb phrase ellipsis, Wijnholds et al. (2019) demonstrated that explicitly copying missing information using non-linear tensor composition models significantly outperforms linear baselines in verb disambiguation.
Mitigating Parameter Explosion via Factorization
The primary obstacle to scaling tensor-based semantics is the "curse of dimensionality." If a word vector has dimension $d$, a matrix requires $d^2$ parameters, and a third-order tensor requires $d^3$. As $d$ scales to capture rich distributional profiles, the number of parameters per relational word explodes exponentially.
To make these models tractable, researchers employ two primary strategies: regression learning and tensor factorization.
Staged and Regression-Based Learning
Instead of learning tensors in isolation, models can learn them by observing how phrases behave in a corpus. Grefenstette et al. (2013) introduced a multi-step regression technique to learn high-rank tensors, proving that distributional models can scale recursively to handle high-arity structures like ditransitive verbs. To handle semantic noise, Kartsaklis et al. (2014) developed a "disambiguate-then-compose" pipeline, using linear regression to build robust word tensors only after explicitly resolving lexical ambiguity via sense induction.
Low-Rank Tensor Decomposition
Rather than maintaining full-rank tensors, models can compress the interaction space into a low-rank format.
- Universal Composition Engines: Bride et al. (2015) introduced a Generalised Lexical Function (GLF) that utilizes a single, shared 3rd-order tensor to transform a modifier's vector into a linear operator. This avoids the need to allocate unique parameters for every individual word.
- CP Decomposition: Fried et al. (2015) applied CANDECOMP/PARAFAC (CP) decomposition to transitive verbs, reducing the parameter footprint by two orders of magnitude while matching or exceeding full-rank models at resolving lexical ambiguity.
- Tucker Decomposition: Frandsen et al. (2019) utilized a 3rd-order composition tensor combined with Tucker decomposition to model phrase composition. Their model applies the tensor to word vectors to generate a precise semantic correction term that is added to the standard vector sum, capturing non-linear syntactic interactions without parameter bloat.
Multi-Dimensional Semantic Spaces
Beyond sentence composition, organizing linguistic data into multi-dimensional tensors allows models to isolate and fuse distinct facets of meaning that matrices collapse together.
By preserving three-way relationships between word pairs and their specific syntactic links, the Structured Distributional Semantic Model (SDSM) developed by Goyal et al. (2013) and the thematic fit framework by Greenberg et al. (2015) construct specialized vector spaces where the exact nature of a syntactic link is preserved.
Similarly, this multi-dimensional framing allows for the simultaneous integration of conflicting semantic relationships. Yih et al. (2013) utilized 3D tensors to model diverse relations like synonymy and antonymy concurrently, preventing the semantic blending that often confuses traditional vector spaces. To merge entirely different data modalities, Jingwei Zhang et al. (2014) applied Bayesian Probabilistic Tensor Factorization (BPTF) to a tensor structured across vocabulary and distinct semantic perspectives (such as dictionary definitions and statistical text patterns), learning a unified latent vector for each word. Finally, Mingda Zhang et al. (2020) extended this to sequential context by using a multi-dimensional interaction tensor to map relationships between context and candidate tokens across multiple semantic categories for story completion.
Key Foundational Work
Experimental Support for a Categorical Compositional Distributional Model of Meaning (2011) — contributed 2 concept elements to this direction.
Distributional Memory: A General Framework for Corpus-Based Semantics (2010) — contributed 2 concept elements to this direction.
Nouns are Vectors, Adjectives are Matrices: Representing Adjective-Noun Constructions in Semantic Space (2010) — contributed 1 concept elements to this direction.