Compositional Distributional Semantics

81 concepts72 papers1 proposal

Distributional semantics has proven remarkably effective at capturing the meanings of individual words. By looking at the contexts in which a word appears, algorithms can map words to dense vectors where geometric closeness corresponds to lexical similarity. In this paradigm, "cat" and "dog" end up near one another because they share similar environments.

However, human language does not stop at individual words. We combine words into phrases and sentences to express an infinite variety of complex ideas. This creates a fundamental challenge: how do we derive the vector representation of a multi-word phrase from the vectors of its constituent parts? This problem lies at the heart of Compositional Distributional Semantics.

The Compositionality Gap

The simplest way to build a phrase representation is through linear vector arithmetic, such as adding or averaging word vectors. While computationally trivial, this approach treats a phrase as a "bag of words," completely ignoring word order and syntactic structure.

As Giesbrecht points out, a simple bag-of-words approach cannot distinguish between "Peter kicked the ball" and "The ball kicked Peter," despite their vastly different meanings. This inability of basic vector operations to capture structural and syntactic dependencies is known as the compositionality gap (Dinu et al.).

Early investigations into bridging this gap focused on element-wise mathematical operations. Mitchell et al. systematically compared different composition functions and demonstrated that element-wise multiplication often outperforms simple addition because it models a form of semantic interaction between words.

Yet, multiplicative composition has its own strict limitations. As phrases grow longer or more complex, element-wise multiplication tends to dilute semantic meaning, causing higher-order representations to lose accuracy (Ng et al.). In fact, Dinu et al. observed that for longer sentences, sophisticated vector aggregation methods frequently underperform simple word-overlap baselines, highlighting how difficult it is for simple vector spaces to retain structural nuance.

Words as Functions: Matrix and Tensor Models

To better mimic the behavior of natural language, researchers began looking toward formal mathematical logic, where certain words act as functions that modify others. Instead of representing every single word as a static vector in the same space, what if some words were matrices or tensors?

Standard Vector Space:
[Adjective Vector] + [Noun Vector] = [Phrase Vector]  (Lacks structural asymmetry)

Functional Space:
[Adjective Matrix] × [Noun Vector] = [Phrase Vector]  (Adjective acts as a function)

This functional perspective was popularized by Baroni et al. through their "Nouns are Vectors, Adjectives are Matrices" framework. In this model, a noun like "car" is a static vector, but an adjective like "red" is a matrix. When you multiply the "red" matrix by the "car" vector, the matrix acts as a linear transformation, mapping the noun to a new position in the vector space that represents "red car."

This algebraic approach scales up to handle more complex syntactic structures:

  • Verbs as Tensors: Polajnar extended this logic to transitive verbs by modeling them as 3rd-order tensors. The verb acts as a multi-argument function that ingests subject and object noun vectors to output a sentence-level representation.
  • Multi-way Interactions: Van de Cruys et al. introduced tensor factorization models to capture the latent, multi-way interactions within Subject-Verb-Object triples, allowing the meaning of the verb to shift dynamically based on its context.
  • Asymmetric Roles: Dima applied a functional approach to compound words (such as German compound nouns), using a masking mechanism where words learn entirely different filters depending on whether they function as the modifier or the head of the phrase.

By moving from vectors to higher-order tensors, these models successfully capture the asymmetric ways that words constrain and modify each other in natural language.

Alternative Geometries for Emergent Meaning

Standard Euclidean vector spaces struggle with emergent meaning—idioms or phrases whose collective meaning is entirely different from the sum of their parts (e.g., "ivory tower" or "green room"). To capture these non-linear interactions, some researchers have abandoned traditional vector spaces altogether in favor of alternative mathematical geometries.

Quantum-Inspired Models

One prominent direction borrows the mathematical machinery of quantum mechanics. Blacoe and Li et al. proposed representing words as density matrices or complex-valued vectors in a Hilbert space.

By using complex numbers, these models can simulate semantic "interference." Just as waves in physics can reinforce or cancel each other out, the interaction between complex word vectors can create non-linear shifts in meaning, allowing the model to represent idiomatic expressions that defy simple addition.

Probabilistic Spaces

Another approach reimagines words and sentences not as fixed points in space, but as regions of uncertainty. Gimpel et al. explored representing sentences as probability distributions.

In this framework, words act as linear operators that transform a Gaussian distribution. This allows the model to naturally capture semantic specificity and entailment: a general phrase covers a wide, high-variance distribution, while a specific phrase narrows the distribution down to a precise region.

Evaluating the Compositionality Gap

How do we measure whether a model truly understands how to compose words, or if it is simply relying on superficial word-matching?

To address this, the NLP community developed specialized benchmarks. Bentivogli et al. introduced the SICK (Sentences Involving Compositional Knowledge) dataset, which forces models to predict semantic relatedness and textual entailment based entirely on structural composition rather than lexical overlap. Similarly, Asaadi et al. created the BiRD dataset to provide a fine-grained analysis of how modifiers shift the semantic weight of bigrams.

Surprisingly, the compositionality gap remains a major hurdle even for modern deep learning architectures. Yu et al. evaluated state-of-the-art Transformer models (like BERT) on controlled phrase-level tasks where simple word-matching was rendered useless (e.g., distinguishing "law school" from "school law").

The authors discovered that despite their massive parameter counts and attention mechanisms, Transformers are surprisingly poor at true phrasal composition. Their high performance on standard benchmarks is largely driven by lexical heuristics rather than a robust understanding of phrase structure.

This revelation underscores why compositional distributional semantics remains a vital area of research: capturing the infinite productivity of human language requires architectures that explicitly model how structure transforms meaning.

Key Foundational Work

A Structured Vector Space Model for Word Meaning in Context (2008) — contributed 1 concept elements to this direction.
Vector-based Models of Semantic Composition (2008) — contributed 1 concept elements to this direction.
Using X-Calculus to Represent Meanings in Logic Grammars (1983) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_compositionality_4780877a,
  title = {semantic compositionality},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-compositionality/rdr_4780877ac7184ee58925f39fd0ee47c4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_compositionality_4780877a_v1,
  title = {semantic compositionality},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-compositionality/rdr_4780877ac7184ee58925f39fd0ee47c4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake