Unraveling the Localized Latents: Learning Stratified Manifold Structures in LLM Embedding Space with Sparse Mixture-of-Experts

Xin Li, Anand D. SarwatearXiv 2025 · DOI · arXiv

Abstract

Large Language Model (LLM) embeddings are typically assumed to lie on a single, smooth global manifold. This paper argues that they actually form a "stratified space" composed of multiple sub-manifolds of varying dimensions. We introduce a novel unsupervised Mixture-of-Experts (MoE) framework using dictionary learning to cluster and analyze frozen LLM embeddings. Our findings reveal that LLM latent spaces are highly non-uniform, with different data domains naturally routing to distinct geometric strata, offering new insights into model interpretability and latent space geometry.

Introduction

LLMs generate rich, high-dimensional embeddings that capture deep semantic relationships. When analyzing these latent spaces, researchers often assume the data lies on a single, continuous global manifold. However, real-world data is heterogeneous—different domains, such as short tweets versus long news articles, possess vastly different complexities and perplexities.

If the embedding space is actually a stratified space—a union of multiple lower-dimensional local manifolds (strata) rather than one smooth surface—then treating it as a single manifold limits our ability to understand how LLMs organize concepts. This work proposes a framework to explicitly test for, discover, and analyze these localized, stratified structures within the embedding spaces of popular LLMs.

Methodology

Instead of using MoE to scale model capacity during training, we repurpose the MoE architecture as an unsupervised analysis tool applied to fixed LLM embeddings (specifically, the last hidden state).

The proposed unsupervised MoE framework for analyzing LLM embedding spaces through dictionary learning experts.

Dictionary Learning Experts

The "experts" in this MoE are Dictionary Learning algorithms implemented via Learnable ISTA (LISTA). Crucially, each expert is hard-coded with a different sparsity level. This allows the model to associate different geometric complexities with different experts.

Attention-Based Gating

An attention-based soft-gating network routes each embedding to the most suitable expert(s). It projects the input into a query space, compares it against learnable key vectors for each expert, and outputs routing probabilities via softmax. By observing which sparsity-constrained expert an embedding is routed to, the framework effectively partitions the embedding space into specialized sub-manifolds (strata).

Experimental Setup

To test how different domains behave in the latent space, we utilized a custom multi-domain dataset including IMDB, Rotten Tomatoes, Amazon Polarity, GLUE/SST2, TweetEval, and AG News. We evaluated embeddings generated by a range of models, including BERT, RoBERTa, GPT-3, Llama-3.2-1B, and DeepSeek-R1-Distill-Qwen-1.5B.

Key Results and Discussion

Our analysis confirms that the embedding space is definitively non-uniform, consisting of varying intrinsic dimensions and complexities depending on the data domain.

3D visualization of data clusters colored by domain label, demonstrating that geometric strata naturally align with semantic domains.

Stratification and Model Scale

Larger models (like Llama-3 and DeepSeek) exhibit a much clearer separation of domains into distinct strata compared to smaller models like BERT. Bigger models also show highly concentrated expert selection, implying their embedding spaces have more uniform perplexity within specific domains. Conversely, smaller models spread assignments across experts, suggesting a "rougher" latent space.

Semantic Alignment

Mathematically derived strata naturally align with human-understandable semantic domains. While the MoE model clustered the geometry blindly, those geometric clusters mapped cleanly back to the original dataset sources. Using PCA to capture 75% of the data variance within each learned stratum, we proved that dimensionality fluctuates significantly (e.g., in Llama-3.2-1B, intrinsic dimensions ranged from 2 to 16 across identified strata).

Conclusion

This work provides empirical evidence that LLM embedding spaces are stratified manifolds. While the current findings are strong, future work will focus on rigorous mathematical and topological proofs of these structures and exploring how stratification can be leveraged to improve LLM performance, architectural design, and interpretability.

Summary updated

BibTeX
@misc{li2025unraveling,
  author = {Xin Li and Anand D. Sarwate},
  title = {{Unraveling the Localized Latents: Learning Stratified Manifold Structures in LLM Embedding Space with Sparse Mixture-of-Experts}},
  year = {2025},
  eprint = {2502.13577},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.13577},
}
Figure