Decomposing High Dimensional Embeddings into Interpretable Concepts

755 concepts594 papers8 proposals

Modern deep learning models process and understand the world through dense, continuous, high-dimensional vector spaces. Whether representing a word, an entire document, an image, or a user's behavioral history, these embeddings are remarkably effective at capturing complex semantic relationships.

However, this effectiveness comes at a steep cost: geometric opacity. In a standard distributed representation, features are "entangled"—smeared across hundreds of continuous dimensions. As noted by Zhiyuan Liu et al. in their comprehensive synthesis, Representation Learning for Natural Language Processing, individual dimensions in these spaces lack human-interpretable meaning. This makes it incredibly difficult to diagnose model errors, audit for safety, or explain why a specific prediction or retrieval decision was made.

To bridge this interpretability gap, a prominent research direction focuses on decomposing these opaque, continuous vector spaces into sparse, discrete, and human-readable components.

The Linear Superposition Hypothesis and Dictionary Learning

The core intuition behind embedding decomposition is that high-dimensional hidden states can be modeled as a linear superposition of discrete semantic factors. Even though an embedding space is dense, the actual concepts it represents are often sparse; any single piece of data only invokes a tiny fraction of all possible human concepts.

By leveraging sparse dictionary learning, researchers can project dense vectors onto a massive, overcomplete dictionary of human-readable concepts. In Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors, Yubei Chen et al. demonstrated that contextualized embeddings can be broken down into a sparse set of "transformer factors." This decomposition reveals a clear internal logic, showing how models progressively build representations from simple word-sense disambiguation in early layers to complex sentence patterns in later layers.

Dense Embedding (Opaque) ───► [ Dictionary Projection ] ───► Sparse Linear Combination (Interpretable)
  [0.12, -0.45, 0.89, ...]                                      0.85 × "zebra" + 0.42 × "savannah"

This approach is highly effective for auditing foundational multimodal models. For instance, Usha Bhalla et al. introduced SpLiCE in Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE). SpLiCE decomposes dense CLIP embeddings into sparse linear combinations of 15,000 explicit textual concepts. By doing so, an abstract image vector is translated into a sparse set of weighted words (e.g., mapping a zebra image to tokens like "zebra" and "africa"). This decomposition allows practitioners to surgically edit model behavior and detect spurious correlations without sacrificing downstream performance.

Grounding Spaces in Sparse Vocabularies

Rather than decomposing dense vectors post-hoc, another facet of this research builds interpretability directly into the architecture by forcing models to embed data into a sparse, vocabulary-grounded space from the start.

In cross-modal retrieval, Bowen Zhang proposed STAIR: Learning Sparse Text and Image Representation in Grounded Tokens. Instead of mapping images and text to abstract dense vectors (like CLIP), STAIR maps them directly to a sparse, high-dimensional token space based on a standard text vocabulary. This allows for direct inspection of what the model sees, making it easy to identify and debug dataset biases—such as the unintended activation of technical metadata tokens from web-crawled training data.

A similar mechanism drives efficiency and transparency in industrial search engines. Dense retrieval models often function as black boxes, making it difficult for engineers to diagnose poor search rankings. To solve this, Zhe Lin et al. designed the Deep Bag-of-Words Model: An Efficient and Interpretable Relevance Architecture for Chinese E-Commerce. By encoding queries and products into sparse word-weight pairs, the architecture achieves deep semantic matching while remaining fully debuggable. As Franco Maria Nardini outlines in Learned Sparse Representations of Text: A Story of Indexing and Retrieval, this paradigm shift combines the rich contextual awareness of modern language models with the clear, auditable mechanics of traditional keyword indexing.

Subnetwork Masking and Disentanglement

When multiple concepts are entangled within the same representation, another way to achieve interpretability is to structurally isolate them. Instead of altering the embedding vectors themselves, we can isolate the specific computational pathways that produce them.

In Disentangling Representations of Text by Masking Transformers, Jan-Willem van de Meent et al. showed that distinct factors of variation (such as a text's sentiment versus its topic) are already separated into latent subnetworks within pretrained models. By learning binary masks to isolate these subnetworks, the authors successfully decoupled entangled features without ever modifying the original model weights, providing a non-destructive method for auditing model fairness and robustness.

Enforcing Discrete and Symbolic Bottlenecks

A radical but highly effective way to ensure interpretability is to replace continuous vector bottlenecks entirely with discrete text tokens or symbolic rules. If a model is forced to communicate using discrete language, its internal reasoning becomes explicitly readable.

This strategy has been successfully applied to prompt engineering and information retrieval:

Probing and Hierarchical Exploration

When an embedding space cannot be altered or discretized, researchers rely on post-hoc probing and interactive visualization to map its internal geometry.

A foundational methodology, detailed in Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks, involves testing fixed-length vectors against simple auxiliary tasks (such as predicting sentence length or word order). This reveals exactly which low-level linguistic properties are preserved or lost during the encoding process.

To make these abstract spaces navigable for humans, interactive tools translate vector proximities into visual relationships. Benjamin Hoover et al. developed exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models, which pairs attention visualizations with real-time similarity searches across large annotated corpora, allowing users to see exactly what a hidden state "thinks" it has found.

For structured data, where embeddings entangle multiple categorical attributes, Yan Zheng et al. introduced EmbeddingTree: Hierarchical Exploration of Entity Features in Embedding. EmbeddingTree hierarchically maps raw entity features back onto the learned embedding space, enabling practitioners to see which features dominate specific vector clusters and clearly identifying where an embedding space fails to capture critical real-world nuances.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 612 papers
  2. Version 1 · 2026-10-04 · 764 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_embedding_interpretability_representatio_4d4f4acf,
  title = {embedding interpretability (representation interpretability)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/embedding-interpretability-representation-interpretability/rdr_4d4f4acf3e7e4b809e4d978f315a8c71},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_embedding_interpretability_representatio_4d4f4acf_v2,
  title = {embedding interpretability (representation interpretability)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/embedding-interpretability-representation-interpretability/rdr_4d4f4acf3e7e4b809e4d978f315a8c71/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake