Geometric Containment Embeddings for Hierarchical Reasoning

18 concepts12 papers1 proposal

Standard representation learning relies heavily on point vectors. In a typical embedding space, an entity or concept is mapped to a single coordinate, and semantic similarity is measured by spatial proximity, such as Euclidean distance or cosine similarity. While this paradigm excels at capturing symmetric relationships (e.g., cat is similar to dog), it fundamentally struggles with asymmetric, hierarchical relationships (e.g., cat is a subclass of mammal).

Points possess no internal volume. Consequently, expressing that one concept encompasses another requires auxiliary relational operators or complex distance thresholds. Furthermore, point representations lack a built-in mechanism for transitivity: if a model learns that $A$ is a subclass of $B$, and $B$ is a subclass of $C$, it must still be explicitly trained or regularized to ensure $A$ is close to $C$.

To resolve this tension, a powerful paradigm shift has emerged: mapping concepts to geometric volumes rather than points. By representing concepts as regions—such as $n$-dimensional spheres or hyper-rectangles—hierarchical subsumption can be modeled directly through spatial containment. If concept $A$ is a sub-category of concept $B$, the geometric volume of $A$ is optimized to reside entirely inside the volume of $B$. This spatial arrangement inherently guarantees transitivity as a mathematical property of the geometry: if volume $A$ is inside volume $B$, and volume $B$ is inside volume $C$, then volume $A$ is automatically inside volume $C$.

Point Embeddings (No Volume)         Volume Embeddings (Containment)
      • A                                     +---------------+
                                              | C             |
            • B                               |   +-------+   |
                                              |   | B     |   |
                  • C                         |   |   • A |   |
                                              |   +-------+   |
                                              +---------------+
Asymmetric hierarchy is ambiguous.     Transitivity is mathematically guaranteed.

Spherical Volumes: Modeling Subsumption with Centers and Radii

The most mathematically straightforward way to give an embedding volume is to define it as an $n$-dimensional sphere (or $n$-ball). A spherical embedding is parameterized by a center vector $c \in \mathbb{R}^d$, which dictates its position in the semantic space, and a scalar radius $r \in \mathbb{R}^+$, which represents the scope or generality of the concept. A larger radius signifies a broader, more abstract category (like Animal), while a smaller radius denotes a highly specific category (like Siamese Cat).

In Differentiating Concepts and Instances for Knowledge Graph Embedding, Lei Hou et al. leverage this geometry to distinguish between abstract concepts and concrete instances. Concepts are modeled as spheres, while instances remain as points. A membership relation (e.g., Alice is a Professor) is satisfied simply by constraining the instance's point vector to fall within the boundaries of the concept's sphere. For category hierarchies, the authors enforce strict physical containment between spheres. For a subclass sphere $A$ to be fully contained within a parent sphere $B$, the distance between their centers plus the radius of $A$ must not exceed the radius of $B$:

$$\|c_A - c_B\|_2 + r_A \le r_B$$

This inequality creates a strong geometric constraint during training. Achille Fokoue et al. apply a similar principle in EmbedS: Scalable, Ontology-aware Graph Embeddings, nesting specific concept volumes inside broader superclass volumes to ensure that taxonomies inherit properties naturally across levels without requiring explicit training for every transitive link.

This geometric formulation has proven highly effective for formal description logics, such as $\mathcal{EL}^{++}$. In EmEL++: Embeddings for EL++ Description Logic, Sutapa Mondal et al. design loss functions that directly translate logical expressions into geometric constraints on $n$-balls. For instance, if the ontology dictates that every Surgeon is a Doctor, the optimization objective penalizes any configuration where the Surgeon sphere escapes the Doctor sphere. As demonstrated by Gunjan Singh et al. in Neuro-Symbolic Techniques for Description Logic Reasoning (Student Abstract), this continuous geometric reasoning scales significantly better than traditional symbolic search trees on massive biomedical ontologies. To handle more complex, many-to-many relationships within these logical frameworks, Biswesh Mohapatra et al. introduce EmELvar: A NeuroSymbolic Reasoner for the EL++ Description Logic, which incorporates a learned variance parameter to allow the boundaries of these spheres to deform elastically depending on the relational context.

While Euclidean spheres work well for flat or moderately branching hierarchies, tree-like structures expand exponentially with depth. To accommodate this exponential growth, Ryota Suzuki et al. introduce Hyperbolic Disk Embeddings for Directed Acyclic Graphs. By defining "formal disks" within a hyperbolic space rather than a Euclidean one, the authors exploit the fact that hyperbolic volume grows exponentially with the radius. This allows the model to seamlessly embed complex directed acyclic graphs (DAGs) with massive numbers of concurrent ancestors and descendants without collapsing the representation space.

Box Embeddings: Capturing Intersections and Lattices

While spheres are computationally efficient, they suffer from a major geometric limitation: the intersection of two spheres is not a sphere. If a concept is defined by the intersection of two parent concepts (e.g., a Bioinformatics Professor is both a Biologist and a Computer Science Professor), spherical embeddings cannot represent this new joint concept space with the same geometric primitive.

To overcome this, researchers introduced box embeddings, which represent nodes as $n$-dimensional hyper-rectangles. A box is parameterized by two vectors: a coordinate-wise minimum $\mathbf{x}_{min}$ and a coordinate-wise maximum $\mathbf{x}_{max}$. The intersection of two hyper-rectangles along orthogonal axes is mathematically guaranteed to yield another hyper-rectangle (or an empty set). This property makes boxes ideal for modeling complex intersection logic and lattice structures.

In Capacity and Bias of Learned Geometric Embeddings for Directed Graphs, Michael Boratko et al. analyze the capacity of these geometric regions. They demonstrate that hyper-rectangular containment provides an optimal inductive bias for directed graphs, as coordinate-wise independent containment perfectly mirrors the properties of partial orders.

However, real-world graphs are rarely perfect hierarchies; they often contain noisy loops or cyclic dependencies. To address this, Dongxu Zhang et al. develop a framework in Modeling Transitivity and Cyclicity in Directed Graphs via Binary Code Box Embeddings. By combining box geometries with binary codes, their model can capture strict transitive flow via containment while simultaneously allowing overlapping regions to represent cyclic shortcuts and local loops.

       Spherical Intersection                  Box Intersection
             (Invalid Shape)                    (Valid Box Shape)
               ___     ___                        +-------+---+
              /   \   /   \                       | A     |   |
             |     |X|     |                      |   +---+---+
              \___/   \___/                       |   | A∩B   |
                                                  +---+-------+
                                                      | B     |
                                                      +-------+

Because the transitivity bias in box embeddings is a structural property of the geometry itself, models require far less data to learn a coherent hierarchy. In Learning Representations for Hierarchies with Minimal Support, Benjamin Rozonoyer et al. demonstrate that by identifying a "minimal support" set of positive and negative relationships and pairing it with box embeddings, models can successfully converge on massive taxonomies using up to 99% less training data than standard point-based sampling methods. The geometric constraints force the unobserved intermediate nodes into their correct topological positions automatically.

Broadening the Scope: Ontologies and Language Models

The utility of spatial containment extends far beyond static taxonomies. In Temporal Knowledge Graph Completion Using Box Embeddings, Johannes Messner et al. extend box volumes to dynamic environments. By treating temporal relations as volumes, the model learns time-dependent subsumption rules—implicitly capturing that a specific short-term relationship is a bounded case of a broader, long-term historical trend through the physical intersection of their regions.

As surveyed by Jiaoyan Chen et al. in Ontology Embedding: A Survey of Methods, Applications and Resources, mapping logical concepts to high-dimensional shapes like balls, boxes, or cones has become the standard blueprint for bridging symbolic reasoning and deep learning.

Intriguingly, this geometric organization may not just be a design choice we impose on models, but an emergent property of language learning itself. In The Lattice Representation Hypothesis of Large Language Models, Bo Xiong investigates how large language models (LLMs) store hierarchical concepts. The author presents evidence that LLMs implicitly organize semantic tokens into nested geometric regions within their hidden states. When an LLM infers that a Golden Retriever is an Animal, it is likely performing a category inclusion task by evaluating the degree of spatial overlap between activation regions in its high-dimensional representation space.

Key Foundational Work

Differentiating Concepts and Instances for Knowledge Graph Embedding (2018) — contributed 3 concept elements to this direction.
EmbedS: Scalable, Ontology-aware Graph Embeddings (2018) — contributed 2 concept elements to this direction.
Hyperbolic Disk Embeddings for Directed Acyclic Graphs (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_box_embeddings_spherical_class_embedding_5c73bbd0,
  title = {box embeddings (spherical class embeddings)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/box-embeddings-spherical-class-embeddings/rdr_5c73bbd0ccd044c6a12de82f7fddcf41},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_box_embeddings_spherical_class_embedding_5c73bbd0_v1,
  title = {box embeddings (spherical class embeddings)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/box-embeddings-spherical-class-embeddings/rdr_5c73bbd0ccd044c6a12de82f7fddcf41/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake