EmbeddingTree: Hierarchical Exploration of Entity Features in Embedding

Yan Zheng, Junpeng Wang, Chin-Chia Michael Yeh, Yujie Fan, Huiyuan Chen, Liang Wang, Wei Zhang

Abstract

High-dimensional embeddings are essential for modern machine learning but often function as "black boxes," where the relationship between specific dimensions and real-world features is obscured. We introduce EmbeddingTree, a framework designed to interpret these embeddings by mapping raw entity features to the learned vector space hierarchically. Using a novel splitting algorithm based on Gaussian Mixture Models and the Bayesian Information Criterion, EmbeddingTree identifies localized feature importance and uncovers hidden data semantics. We demonstrate the tool's effectiveness on large-scale merchant and music datasets, providing a path for debugging and refining complex embedding models.

Introduction

In machine learning, entities such as customers, merchants, or songs are represented as high-dimensional vectors. While effective for downstream tasks like recommendation, these embeddings "entangle" multiple features, making it difficult to understand why specific entities cluster together. Existing "parallel exploration" tools often fail to capture the hierarchical nature of real-world data—for instance, a merchant's location might be the primary determinant of its embedding, while its business category only becomes relevant within a specific geographic context. EmbeddingTree addresses this gap by providing a localized, hierarchical view of feature importance.

Methodology

EmbeddingTree combines a recursive splitting algorithm with an interactive visualization system to bridge the gap between raw metadata and latent embeddings.

The Embedding-BIC Algorithm

To construct the hierarchy, the algorithm recursively partitions the embedding space. For each available entity feature (e.g., "City == New York"), it evaluates how effectively that feature splits the current cluster into two distinct Gaussian distributions. This is achieved using a Gaussian Mixture Model (GMM) approach combined with the Bayesian Information Criterion (BIC). The feature yielding the highest log-likelihood for the split is selected as the node, ensuring that each branch represents a statistically significant differentiation in the embedding space.

Interactive Visualization System

The framework provides three coordinated views to facilitate exploration:

  1. Tree View: Visualizes the hierarchy of features defining the embedding structure.
  2. Dimensionality Reduction View: Displays the entity "cloud" using techniques like PCA, t-SNE, or UMAP.
  3. Data Table: Offers granular access to raw metadata for selected entities.
The EmbeddingTree interface featuring the Tree View (a), Dimensionality Reduction View (b), and Data Table (c).

Case Studies and Key Insights

The researchers evaluated EmbeddingTree on two diverse datasets: a proprietary merchant dataset from Visa (70 million merchants) and the public 30Music dataset (~45,000 users).

Localized Feature Importance

A primary finding is that feature significance is context-dependent. In the merchant data, the tree revealed that for merchants in Los Angeles, the Merchant Category Code (MCC) was the dominant differentiator. Conversely, for merchants in Manhattan, transaction frequency played a more critical role. This localized insight is often lost in global visualization tools.

Visualization of embedding clusters dominated by different features such as Location, MCC, and Frequency.

Identifying Inconsistencies and Hidden Semantics

EmbeddingTree helps identify "inconsistent" embeddings where clusters exist that cannot be explained by known features. In one instance, the tool isolated two distinct merchant clusters with identical metadata. Further investigation revealed the embedding had captured a hidden feature—the use of "Square" payment hardware—which was not present in the original dataset.

Refining Embedding Models

In the music dataset, the tool highlighted deficiencies in user embeddings. While gender and playlist count were well-represented, "subscription type" (Free vs. Premium) failed to create distinct splits in the vector space, indicating that the training process did not sufficiently encode this attribute.

Conclusion and Future Work

EmbeddingTree provides a robust method for "cracking the code" of black-box embeddings. While currently used by domain practitioners at Visa, future work includes formal user studies to quantify its effectiveness compared to traditional tools. Additionally, the authors plan to introduce manual refinement capabilities, allowing users to inject domain knowledge by manually pruning or growing branches of the tree.

Summary updated

Abstract

Embedding learning transforms discrete data entities into continuous numerical representations, encoding features/properties of the entities. Despite the outstanding performance reported from different embedding learning algorithms, few efforts were devoted to structurally interpreting how features are encoded in the learned embedding space. This work proposes EmbeddingTree, a hierarchical embedding exploration algorithm that relates the semantics of entity features with the less-interpretable embedding vectors. An interactive visualization tool is also developed based on EmbeddingTree to explore high-dimensional embeddings. The tool helps users discover nuance features of data entities, perform feature denoising/injecting in embedding training, and generate embeddings for unseen entities. We demonstrate the efficacy of EmbeddingTree and our visualization tool through embeddings generated for industry-scale merchant data and the public 30Music listening/playlists dataset.
BibTeX
@inproceedings{zheng2023embeddingtree,
  author = {Yan Zheng and Junpeng Wang and Chin-Chia Michael Yeh and Yujie Fan and Huiyuan Chen and Liang Wang and Wei Zhang},
  title = {{EmbeddingTree: Hierarchical Exploration of Entity Features in Embedding}},
  booktitle = {PacificVis},
  year = {2023},
}
Figure