TL;DR
Researchers have developed a way to visualize Large Language Model (LLM) behavior by mapping text generation onto a "vocabulary network" built from token embeddings. Their findings reveal that generation is not a simple walk between similar words, but a complex leap across a fragmented, scale-free graph—a perspective that could help detect and prevent model "degeneration" (repetitive loops) in real-time.
What Problem Does This Paper Address?
Despite their ubiquity, LLMs remain "black boxes." We understand the math of the Transformer architecture, but we lack a "microscopic" understanding of how a model navigates its vast vocabulary during inference. Specifically, it is unclear how the static geometry learned during pre-training (the embeddings) relates to the dynamic process of generating text.
Existing explainability methods often treat the model as a whole or look at internal activations, but they rarely reference the actual "space" where tokens live. This paper seeks to bridge that gap by treating the vocabulary as a physical network and observing how the model "dances" across it.
Methods
The authors constructed a token-similarity network for six open-source LLMs.
- Network Construction: Every token in the vocabulary is a node. An edge is drawn between two nodes if the cosine similarity of their input embeddings exceeds a threshold $S$.
- Reference Cutoff: To make models comparable, they set a threshold $S_c$ where the mean degree of the network is exactly 1 (the point where a "giant component" typically begins to emerge in random graphs).
- Logit Lens: They used the "logit lens" technique to project the hidden states of every internal layer back into the vocabulary space. This allowed them to identify the "settling layer"—the earliest point in the model's depth where it commits to the final output token.
- Mapping Generation: By treating the generated sequence as a series of directed steps on this static graph, they could visualize whether the model stays in a "neighborhood" or jumps between distant clusters.
Datasets
The study analyzed six prominent open-source models:
- Meta-Llama-3.1-8B (Pre-trained base)
- Qwen3-8B (Post-trained)
- DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B (Distilled reasoning models)
- DeepSeek-R1-0528-Qwen3-8B
- gpt-oss-20B (Mixture-of-Experts reasoning model)
Evaluation was performed using nine prompts across three distinct domains: metallic glasses (materials science), machine learning, and movies.
Key Insights and Ideas
- The Vocabulary is an Archipelago: Unlike random embeddings, trained LLM embeddings form a highly fragmented network. It is "scale-free" (a few hubs have many connections, most have few), but these hubs are often punctuation or formatting tokens rather than semantic concepts.
- Generation is Not a Walk: Surprisingly, consecutive tokens in a sentence are rarely neighbors in the similarity network. In fact, they are often less likely to be neighbors than tokens chosen at random from the same text. This implies that the attention mechanism is constantly pulling the model across different "islands" of the vocabulary.
- Distillation is an Inheritance: Distilling a model (like DeepSeek-R1 into Llama) drastically changes its reasoning capabilities but barely moves its input embeddings. The "knowledge space" is largely fixed during the initial pre-training phase.
- Degeneration as Confinement: When a model starts repeating itself (hallucinating or looping), it physically collapses into a small, densely connected sub-graph. This "spatial collapse" can be measured, offering a new way to monitor model health during inference.
Key Results
- Power-Law Topology: All models exhibited a degree distribution $P(k) \sim k^{-\gamma}$ with $\gamma \approx 2.19$, similar to word co-occurrence networks in natural language.
- Late Settling: In most models, the final token identity only "settles" very late in the model's depth (median depth $\sim 0.87$ of the total layers). Base models settle earlier than reasoning-tuned models.
- Domain Specificity: 70% of tokens generated for a specific domain (e.g., metallic glasses) are never used in other domains. This suggests that models could potentially be "pruned" into domain-specific versions by identifying these functional sub-graphs.
- Repetition Metrics: The authors found a strong negative correlation ($r = -0.98$) between the "token-revisit rate" on the graph and the diversity of the generated text, providing a quantitative metric for "trapped" generation.
Limitations and Future Work
The authors acknowledge that their current network is built on static input embeddings. In reality, Transformers use dynamic attention that changes the "effective" similarity between tokens based on context. Future work aims to incorporate these context-dependent attention scores into the graph.
Additionally, the study suggests that "strategic path optimization" on these graphs could lead to better decoding algorithms. Instead of just picking the most likely next token, we might steer the model toward under-explored but promising regions of the vocabulary network to improve creativity and reduce hallucination.
Summary updated