Decomposing Latent Embeddings into Sparse Semantic Atoms

This direction is merged into another direction: embedding interpretability (representation interpretability). Its last version stays readable and citable here.

Version 1 of 118 papers21 conceptsfrozen 2026-10-04

Modern machine learning relies on dense latent embeddings—high-dimensional vectors that represent images, words, or concepts. While these "black-box" representations are powerful, they are notoriously difficult for humans to interpret. A single dense vector for the word "bank" must somehow encode both a financial institution and the side of a river, leading to a "conflation problem" where distinct meanings are mashed together in a single point in space.

A growing body of research suggests that these dense vectors can be decomposed into a sparse set of "semantic atoms." By projecting dense embeddings onto a large dictionary of basis vectors, we can represent complex entities as simple, additive combinations of human-readable concepts.

The Geometry of Meaning

The core intuition behind this research is that the information within a dense embedding is not distributed randomly. Instead, it follows a structured geometry that can be recovered through sparse decomposition. Hiroaki Yamagiwa, Hidetoshi Shimodaira, and Momose Oyama demonstrated that by applying Independent Component Analysis (ICA) to embeddings, one can extract sparse semantic axes that remain consistent across different languages and models.

In this view, a specific concept is not just a point, but a sum of independent parts. For example, a specific car brand might be represented as the sum of a "vehicle" axis and a "nationality" axis. This additive structure allows us to move from a "black box" vector to a transparent list of attributes.

Solving Polysemy with Sparsity

One of the primary failures of standard embeddings is polysemy—the fact that one word can have multiple meanings. In a dense space, the vector for "crane" sits somewhere in the middle of "birds" and "construction equipment," potentially satisfying neither context perfectly.

Researchers like Alisa Liu and Doug Downey address this by decomposing aggregate word embeddings into "sense atoms." By representing a word as a sparse linear combination of shared basis vectors, a model can isolate individual meanings. This allows a system to generate distinct natural language definitions for different senses of the same word vector.

Similarly, the SPINE (SParse Interpretable Neural Embeddings) method developed by Anant Subramanian and colleagues uses a denoising autoencoder to transform dense embeddings into high-dimensional sparse representations. In this sparse space, different meanings of a word activate separate, non-overlapping dimensions, preventing the semantic blending that plagues dense models.

Auditing and Editing the Latent Space

If we can decompose an embedding into a sparse set of words, we can "audit" what a model is actually seeing. The SpLiCE (Sparse Linear Concept Embeddings) method, introduced by Usha Bhalla, Alex Oesterling, and Suraj Srinivas, applies this to multimodal models like CLIP.

By solving a non-negative Lasso optimization problem, SpLiCE decomposes a CLIP image embedding into a sparse combination of thousands of potential dictionary concepts. An image of a zebra might be decomposed into "zebra," "Africa," and "grazing." This transparency allows researchers to:

  1. Detect Biases: See if a model is associating "doctor" primarily with "male" concepts.
  2. Identify Spurious Correlations: Determine if a "fish" classifier is actually just looking for "water" or "nets."
  3. Surgical Editing: Modify the model's behavior by zeroing out specific semantic atoms without retraining the entire network.

Grounding Atoms in Human Knowledge

Are these sparse dimensions truly meaningful, or are they just mathematical artifacts? Vanda Balogh and Gábor Berend validated the semantic content of these dimensions by "decoding" them using ConceptNet, a commonsense knowledge base. They found that the words grouped within a single sparse dimension are significantly more semantically related than random sets, proving that sparse architectures naturally align with human-interpretable categories.

This alignment is even useful for cross-lingual tasks. Ndapandula Nakashole developed NORMA, which uses sparse dictionary learning to map word embeddings between distant languages like English and Chinese. By decomposing words into their local semantic neighborhoods before mapping, the system can handle the structural differences between languages that simpler, global alignment methods miss.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decomposing_latent_embeddings_into_spars_19668a5b,
  title = {Decomposing Latent Embeddings into Sparse Semantic Atoms},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decomposing-latent-embeddings-into-sparse-semantic-atoms/rdr_19668a5b8b444ee097c3509eaf125984},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_decomposing_latent_embeddings_into_spars_19668a5b_v1,
  title = {Decomposing Latent Embeddings into Sparse Semantic Atoms},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decomposing-latent-embeddings-into-sparse-semantic-atoms/rdr_19668a5b8b444ee097c3509eaf125984/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}