Traditional word embeddings represent concepts as single points in a high-dimensional vector space. While this approach is excellent at capturing semantic similarity—placing "dog" near "cat"—it fundamentally struggles with the directional nature of human logic. A point has no volume, and therefore no way to "contain" another point. This makes it difficult for models to represent that a "poodle" is a "dog," but not every "dog" is a "poodle."
To solve this, researchers have shifted from point-based vectors to geometric region embeddings. By representing concepts as shapes with volume—such as Gaussians, boxes, or balls—models can use spatial containment to represent logical hierarchies.
The Geometry of Inclusion
In a standard vector space, the relationship between "fruit" and "apple" is represented by the distance between two coordinates. However, as Luke Vilnis and Andrew McCallum argue in Word Representations via Gaussian Embedding, this distance is symmetric. If "apple" is close to "fruit," then "fruit" is equally close to "apple." This symmetry fails to capture lexical entailment (hyponymy), where one term is more specific than the other.
By mapping words to Gaussian distributions instead of points, the model gains a new dimension: variance. A broad, general concept like "entity" can be represented by a distribution with high variance (a large "cloud"), while a specific instance like "Golden Retriever" has low variance (a tight "cluster"). Ben Athiwaratkun and Andrew Gordon Wilson demonstrate in Hierarchical Density Order Embeddings that this allows the density of a general concept to physically encompass the density of a specific one, creating a "soft" hierarchy.
Box Embeddings and Set Logic
While Gaussians provide a probabilistic way to model hierarchy, they can be mathematically difficult to intersect or join in complex logical queries. This led to the development of Box Embeddings (hyperrectangles).
In this framework, an entity is an $n$-dimensional box. The "volume" of the box represents the specificity of the concept. Xuelu Chen and Michael Boratko show in Probabilistic Box Embeddings for Uncertain Knowledge Graph Reasoning that boxes allow for calibrated probabilistic reasoning. If Box A is entirely inside Box B, the model "knows" with 100% certainty that A entails B.
This geometric approach is particularly powerful for Knowledge Graphs. Zijie Huang introduces Concept2Box, which treats high-level categories as boxes and specific instances as vectors. This dual-geometry approach allows the model to navigate the "two-view" nature of knowledge: the abstract ontology and the concrete facts.
Capturing Uncertainty and Ambiguity
Beyond hierarchy, region-based embeddings are naturally suited for modeling uncertainty. In vision-language tasks, a single caption like "a bird in the sky" could refer to thousands of different images.
Yatai Ji and colleagues propose MAP, a modality-agnostic model that represents image regions and text tokens as multivariate Gaussians. Instead of forcing a vague caption to a single point, the model represents it as a region of potential matches. Similarly, Sanghyuk Chun uses Probabilistic Language-Image Pre-Training to show that this probabilistic view significantly improves zero-shot performance by accounting for the inherent ambiguity of how humans describe the world.
Hierarchy in Large Language Models
Even in modern transformer-based models, these geometric structures appear to emerge naturally. Kiho Park and Victor Veitch found in The Geometry of Categorical and Hierarchical Concepts in Large Language Models that LLMs like Llama-3 represent categories as polytopes (complex geometric shapes). They discovered that the internal geometry of these models mirrors human taxonomies like WordNet, with specific sub-categories "packed" into the broader parent concepts through orthogonal directions.
By moving from points to regions, these models gain the ability to represent the world not just as a collection of similar things, but as a structured, nested, and often uncertain system of ideas.