In modern machine learning, representations are dominated by vectors. Whether we are modeling words, entities in a knowledge graph, or entire sentences, the standard approach is to map each concept to a single point in a high-dimensional continuous vector space.
While point-based embeddings excel at capturing semantic similarity via spatial proximity (such as cosine distance), they possess a fundamental geometric limitation: points are zero-dimensional. They have no volume, no boundaries, and no internal structure. This makes them inherently ill-suited for representing hierarchies, taxonomies, and asymmetric logical relationships.
The Flaw of Zero-Dimensional Points
To understand why point embeddings struggle with hierarchies, consider the relationship between the concepts Animal, Dog, and Poodle. This is a directed, asymmetric taxonomy: every poodle is a dog, and every dog is an animal, but the reverse is not true. Furthermore, Animal is a broad, highly general category, whereas Poodle is narrow and specific.
In a traditional vector space, both Animal and Poodle are represented as single points. The distance metric between them is symmetric; the vector space itself has no native mechanism to show that one concept encompasses the other. As noted by Dinesh Garg et al. in Quantum Embedding of Knowledge for Reasoning, point-based representations lack the geometric capacity to model the hierarchical "area" or inclusion relationships inherent in complex taxonomies. Because a point has no spatial extent, it cannot act as a container for other concepts.
The Geometry of Inclusion
To bridge this representation gap, a powerful alternative paradigm has emerged: representing concepts not as points, but as geometric regions with volume. By giving embeddings spatial extent, asymmetric hierarchical relationships can be modeled naturally through set-theoretic operations like containment and intersection.
Box Embeddings (Hyperrectangles)
One prominent approach is to represent concepts as $n$-dimensional boxes (hyperrectangles). In this framework, developed by researchers like Michael Boratko et al. in Word2Box: Capturing Set-Theoretic Semantics of Words using Box Embeddings, the generality of a concept is directly proportional to the volume of its box.
A broad concept like Animal is represented by a massive box that occupies a large portion of the latent space. A specific sub-concept like Dog is a smaller box completely nested inside the Animal box ($Dog \subseteq Animal$). This geometric configuration naturally satisfies the transitivity of hierarchies: if the Poodle box is inside the Dog box, it is mathematically guaranteed to be inside the Animal box.
Spheres and $\mathcal{N}$-balls
Regions can also take spherical forms. For instance, Lei Hou et al. introduced TransC in Differentiating Concepts and Instances for Knowledge Graph Embedding, which differentiates between concepts and instances by modeling high-level concepts as spheres and specific instances as points located inside those spheres.
Taking this further, Tiansi Dong et al. demonstrated in Imposing Category Trees Onto Word-Embeddings Using A Geometric Construction that existing point-based word vectors can be mapped into $n$-dimensional balls ($\mathcal{N}$-balls). This geometric construction allows models to encode strict category trees with 100% precision while preserving the original semantic distances between items.
Probabilistic Regions and Density
Geometric regions do not necessarily require hard boundaries. Instead of solid shapes, concepts can be modeled as continuous probability distributions, where the "volume" corresponds to the spread or variance of the density cloud.
In a probabilistic embedding space, a highly abstract or general term is represented as a wide Gaussian distribution with high variance, spreading its probability mass across a large neighborhood. A specific term is modeled as a tight, concentrated Gaussian with low variance.
Ivan Titov pioneered this perspective with the Bayesian Skip-gram model in Embedding Words as Distributions with a Bayesian Skip-gram Model, demonstrating that modeling words as densities successfully captures semantic uncertainty and levels of generality.
This intuition scales effectively to larger linguistic structures. For example, Kevin Gimpel et al. developed probabilistic sentence representations in Learning Probabilistic Sentence Representations from Paraphrases to automatically capture sentence specificity and entailment, allowing the model to distinguish between broad, general statements and highly precise assertions.
Why It Works: Quantifying Generality
The core reason region-based embeddings excel at hierarchical reasoning is that they allow us to calculate conditional probabilities and entailment scores directly from the geometry of the space.
In a point-based model, calculating whether "Poodle" entails "Dog" requires learning a separate, asymmetric classifier layer on top of the static vectors. In a box embedding space, however, the entailment probability $P(\text{Dog} \mid \text{Poodle})$ is calculated natively as the ratio of the volume of their intersection to the volume of the target concept:
$$P(B \mid A) = \frac{\text{Vol}(A \cap B)}{\text{Vol}(A)}$$
If box $A$ (Poodle) is entirely enclosed within box $B$ (Dog), their intersection volume equals the volume of $A$, resulting in a perfect entailment probability of $1.0$. Conversely, the reverse probability $P(\text{Poodle} \mid \text{Dog})$ will naturally evaluate to a small fraction, because the volume of the Poodle box constitutes only a tiny fraction of the broader Dog box.
As highlighted by Tejas Chheda et al. in Box Embeddings: An open-source library for representation learning using geometric structures, this mathematical property allows models to represent partial orders and varying levels of granularity natively.
When knowledge graphs contain high levels of uncertainty, this volume-based calculation provides a calibrated measure of fact confidence, as shown by Xuelu Chen et al. in Probabilistic Box Embeddings for Uncertain Knowledge Graph Reasoning.
To achieve even stricter mathematical properties, Fred Xu et al. introduced FUSE: Measure-Theoretic Compact Fuzzy Set Representation for Taxonomy Expansion, which uses measure theory to ensure that the learned regions are fully closed under all set operations (intersection, union, and complement), allowing models to perform complex logical reasoning over taxonomies without losing structural integrity.
Key Foundational Work
Hierarchical language models for one-stage speech interpretation (2005) — contributed 1 concept elements to this direction.
How to spot a Dalmatian in a pack of Dogs; A data-driven approach to searching unannotated images using natural language (2007) — contributed 1 concept elements to this direction.