Box Embeddings: An open-source library for representation learning using geometric structures

Tejas Chheda, Purujit Goyal, Trang Tran, Dhruvesh Patel, Michael Boratko, Shib Sankar Dasgupta, Andrew McCallum

Abstract

Traditional vector embeddings represent entities as points in high-dimensional space, which limits their ability to capture hierarchical relationships and varying levels of generality. This paper introduces box-embeddings, a unified open-source library for modeling entities as $n$-dimensional hyper-rectangles. By leveraging geometric regions, this approach naturally represents hierarchies and probabilistic logic. We demonstrate the library's effectiveness on WordNet and MNLI tasks, showing significant improvements in modeling transitive relationships and complex logical entailment.

Introduction

In the current landscape of deep learning, vectors are the primary tool for representation. While effective for calculating similarity, they struggle to capture complex logical structures like hierarchies, partial orders, or varying levels of generality. For instance, representing the concept "Fruit" as more general than "Banana" is non-trivial in point-based vector spaces. Box embeddings address this by representing entities as regions, where the volume of the region correlates with the concept's scope.

Geometric Representation of Boxes

A box embedding represents an entity as an $n$-dimensional rectangle, defined as the Cartesian product of intervals. Each box is characterized by a lower-left coordinate ($z$) and a top-right coordinate ($Z$).

Geometric representation of two boxes, A and B, in a 2D coordinate system.

This geometric structure allows for intuitive modeling of relationships:

  1. Generality: A larger box represents a more general concept.
  2. Hierarchy: Containment (Box A $\subset$ Box B) naturally represents a "sub-type" relationship (e.g., Banana $\subset$ Fruit).
  3. Probabilistic Logic: The volume of a box can represent a marginal probability, while the volume of the intersection of two boxes represents their joint probability.

Modeling Hierarchies and Logical Entailment

The primary advantage of box embeddings is their ability to map semantic hierarchies directly to geometric overlaps. This allows the model to calculate conditional probabilities—$P(A|B)$ is simply the volume of the intersection divided by the volume of Box B.

Mapping a semantic hierarchy to overlapping box regions.

As illustrated above, concepts like "Herb" and "Fruit" can overlap to include "Banana," while "Basil" and "Thyme" are contained within the "Herb" box. This mapping provides a clear visual and mathematical framework for transitive relationships.

The box-embeddings Library

The core contribution of this work is the box-embeddings library, which provides a high-level BoxTensor abstraction for PyTorch and TensorFlow. The library addresses several historical implementation challenges:

  • Numerical Stability: It offers multiple parameterizations (e.g., MinDeltaBoxTensor, SigmoidBoxTensor) to ensure that the "max" corner of a box always stays above the "min" corner during training.
  • Gradient Sparsity: To solve the problem of vanishing gradients when boxes do not overlap, the library implements "soft" operations. By using Gumbel Intersections and Soft Volumes, it creates a smooth loss landscape that pulls disjoint boxes together.
  • Regularization: The authors include specialized L2 side-box regularization to prevent boxes from becoming excessively small or floating in empty space.

Evaluation and Results

The library was evaluated on WordNet (lexical relations) and MNLI (natural language inference) tasks:

  • WordNet Hierarchy: The library replicated state-of-the-art results in modeling transitive closures. Using regularization improved the F1 score for predicting hypernymy from 44.2% to 59.4% in a zero-shot setting.
  • Natural Language Inference: By encoding sentences as boxes, the model achieved 78% accuracy on the MNLI task using a RoBERTa encoder, demonstrating that box embeddings can scale to complex, high-dimensional NLP tasks.

Conclusion

The box-embeddings library provides a robust, framework-agnostic toolkit for geometric representation learning. By moving beyond point-based vectors, researchers can more effectively model the inherent hierarchy and uncertainty of real-world data. Future work involves extending these structures to even more complex shapes and exploring their use in uncertain knowledge graph reasoning.

Summary updated

BibTeX
@inproceedings{chheda2021box,
  author = {Tejas Chheda and Purujit Goyal and Trang Tran and Dhruvesh Patel and Michael Boratko and Shib Sankar Dasgupta and Andrew McCallum},
  title = {{Box Embeddings: An open-source library for representation learning using geometric structures}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=B34Z94ltIbq},
}
Figure