Abstract
Traditional vector embeddings represent entities as points in high-dimensional space, which limits their ability to capture hierarchical relationships and varying levels of generality. This paper introduces box-embeddings, a unified open-source library for modeling entities as $n$-dimensional hyper-rectangles. By leveraging geometric regions, this approach naturally represents hierarchies and probabilistic logic. We demonstrate the library's effectiveness on WordNet and MNLI tasks, showing significant improvements in modeling transitive relationships and complex logical entailment.
Introduction
In the current landscape of deep learning, vectors are the primary tool for representation. While effective for calculating similarity, they struggle to capture complex logical structures like hierarchies, partial orders, or varying levels of generality. For instance, representing the concept "Fruit" as more general than "Banana" is non-trivial in point-based vector spaces. Box embeddings address this by representing entities as regions, where the volume of the region correlates with the concept's scope.
Geometric Representation of Boxes
A box embedding represents an entity as an $n$-dimensional rectangle, defined as the Cartesian product of intervals. Each box is characterized by a lower-left coordinate ($z$) and a top-right coordinate ($Z$).
This geometric structure allows for intuitive modeling of relationships:
- Generality: A larger box represents a more general concept.
- Hierarchy: Containment (Box A $\subset$ Box B) naturally represents a "sub-type" relationship (e.g., Banana $\subset$ Fruit).
- Probabilistic Logic: The volume of a box can represent a marginal probability, while the volume of the intersection of two boxes represents their joint probability.
Modeling Hierarchies and Logical Entailment
The primary advantage of box embeddings is their ability to map semantic hierarchies directly to geometric overlaps. This allows the model to calculate conditional probabilities—$P(A|B)$ is simply the volume of the intersection divided by the volume of Box B.
As illustrated above, concepts like "Herb" and "Fruit" can overlap to include "Banana," while "Basil" and "Thyme" are contained within the "Herb" box. This mapping provides a clear visual and mathematical framework for transitive relationships.
The box-embeddings Library
The core contribution of this work is the box-embeddings library, which provides a high-level BoxTensor abstraction for PyTorch and TensorFlow. The library addresses several historical implementation challenges:
- Numerical Stability: It offers multiple parameterizations (e.g.,
MinDeltaBoxTensor,SigmoidBoxTensor) to ensure that the "max" corner of a box always stays above the "min" corner during training. - Gradient Sparsity: To solve the problem of vanishing gradients when boxes do not overlap, the library implements "soft" operations. By using Gumbel Intersections and Soft Volumes, it creates a smooth loss landscape that pulls disjoint boxes together.
- Regularization: The authors include specialized L2 side-box regularization to prevent boxes from becoming excessively small or floating in empty space.
Evaluation and Results
The library was evaluated on WordNet (lexical relations) and MNLI (natural language inference) tasks:
- WordNet Hierarchy: The library replicated state-of-the-art results in modeling transitive closures. Using regularization improved the F1 score for predicting hypernymy from 44.2% to 59.4% in a zero-shot setting.
- Natural Language Inference: By encoding sentences as boxes, the model achieved 78% accuracy on the MNLI task using a RoBERTa encoder, demonstrating that box embeddings can scale to complex, high-dimensional NLP tasks.
Conclusion
The box-embeddings library provides a robust, framework-agnostic toolkit for geometric representation learning. By moving beyond point-based vectors, researchers can more effectively model the inherent hierarchy and uncertainty of real-world data. Future work involves extending these structures to even more complex shapes and exploring their use in uncertain knowledge graph reasoning.
Summary updated