Decoding in Geometry: Alleviating Embedding-Space Crowding for Complex Reasoning

Yixin Yang, Qingxiu Dong, Zhifang SuiarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces "embedding-space crowding," a phenomenon where LLMs tend to concentrate their next-token predictions on geometrically similar embeddings, hindering complex reasoning. They propose CraEG, a novel sampling method that uses token embedding geometry to mitigate this crowding, leading to improved reasoning accuracy and diversity without requiring additional training.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly used for complex reasoning tasks, relying on autoregressive decoding to generate text. Standard decoding strategies like temperature scaling and top-p sampling adjust the probability distribution of the next token to balance output quality and diversity. However, these methods operate solely on token probabilities and ignore the underlying geometric relationships between token embeddings in the model's latent space. The authors identify a new problem: "embedding-space crowding," where the model's probability mass for the next token becomes concentrated on a small cluster of geometrically similar tokens. This crowding is shown to be negatively correlated with reasoning success, suggesting that existing methods overlook a crucial aspect of decoding that limits performance on complex tasks.

Methods

The core of this paper is the introduction and exploitation of "embedding-space crowding."

  • Quantifying Crowding: The authors define two metrics:
  • Token-Level Crowding Score: Measures how much probability mass is concentrated on tokens geometrically similar to a specific token.
  • Step-Level Crowding Score: An aggregated measure of token-level crowding at a given decoding step, representing the expected crowding across all possible next tokens.
  • Sequence-Level Crowding Score: The average step-level crowding over an entire generated sequence.
  • CraEG (Crowding-Aware Sampling via Embedding Geometry): This is a novel, plug-in sampling method designed to alleviate embedding-space crowding.
  • It identifies an "effective correction set" of high-probability tokens to focus the intervention.
  • It computes a "crowding-aware correction factor" for each token in this set, which downweights tokens that are both high-probability and highly crowded. This factor is designed to be geometry-aware, penalizing tokens that are close in embedding space and have high probability.
  • A "step-level correction strength factor" dynamically adjusts the overall correction magnitude at each decoding step, adapting to the distribution's structure.
  • The method is training-free, requires no additional passes, and can be integrated with standard sampling strategies like top-p and temperature scaling.

Datasets

  • AIME24, AIME25, HMMT25: These are mathematical reasoning benchmarks used for evaluating the model's performance. They are described as having well-defined problems with verifiable numeric answers, making them suitable for assessing complex reasoning capabilities.

Key Insights and Ideas

  • Embedding-Space Crowding is a Real Phenomenon: The paper convincingly demonstrates that LLMs exhibit "embedding-space crowding," where probability mass clusters around geometrically similar tokens. This is a novel observation that goes beyond traditional probability-based analysis.
  • Crowding Hinders Reasoning: There's a statistically significant negative correlation between embedding-space crowding and reasoning success, particularly in mathematical problem-solving. This suggests that when the model "thinks" in a narrow, geometrically constrained way, it's less likely to arrive at the correct answer.
  • Geometry Matters for Decoding: The paper highlights that the geometric structure of token embeddings is not just an artifact of training but actively influences decoding behavior and downstream task performance.
  • CraEG Offers a Geometry-Guided Solution: By explicitly addressing embedding-space crowding using geometric information, CraEG provides a principled way to improve both the accuracy and diversity of generated text, especially for complex reasoning tasks.
  • Crowding and Entropy Capture Different Aspects: The analysis shows that while entropy measures overall uncertainty, crowding captures the organization of probability mass in embedding space, and that crowding is a more significant predictor of reasoning correctness than entropy alone.

Key Results

  • Improved Reasoning Accuracy: CraEG consistently improves Avg@32 (average score across 32 runs) and Pass@8 (fraction of problems solved in at least one of 8 trials) on mathematical reasoning benchmarks across multiple LLMs (Qwen3-1.7B, Qwen3-4B, Hunyuan-1.8B-Instruct). For example, on Qwen3-1.7B with temperature=1.0 and top-p=1.0, Avg@32 improved by 0.52 points and Pass@8 by 1.98 points.
  • Enhanced Diversity: CraEG also leads to gains in diversity metrics like Distinct-n and semantic diversity. For Qwen3-1.7B, Dist-N increased by 1.17 points and semantic diversity by 0.62 points under the same settings.
  • Robustness: The improvements are observed across different decoding configurations (temperature/top-p settings) and models, indicating robustness.
  • Crowding as a Predictor: Logistic regression analysis showed that crowding is a significant negative predictor of correctness, even when controlling for entropy.
  • Ablation Studies: The paper validates the importance of the nonlinear weighting mechanism and the dynamic correction strength factor, showing that a fixed correction strength can harm accuracy while improving diversity.

Limitations and Future Work

  • Model Dependency of Diversity-Accuracy Tradeoff: While CraEG generally improves both accuracy and diversity, the paper notes that the exact tradeoff can be model-dependent.
  • Future Work on Adaptive Weighting: The authors suggest that further characterization of the tradeoff between nonlinear and linear weighting, and the development of adaptive weighting schemes, could be fruitful areas for future research.
  • Broader Applications: While demonstrated on mathematical reasoning, the paper suggests that geometry-aware decoding methods could be beneficial for other complex reasoning tasks.

Summary updated

Abstract

Sampling-based decoding underlies complex reasoning in large language models (LLMs), where decoding strategies critically shape model behavior. Temperature- and truncation-based methods reshape the next-token distribution through global probability reweighting or thresholding to balance the quality-diversity tradeoff. However, they operate solely on token probabilities, ignoring fine-grained relationships among tokens in the embedding space. We uncover a novel phenomenon, embedding-space crowding, where the next-token distribution concentrates its probability mass on geometrically close tokens in the embedding space. We quantify crowding at multiple granularities and find a statistical association with reasoning success in mathematical problem solving. Motivated by this finding, we propose CraEG, a plug-and-play sampling method that mitigates crowding through geometry-guided reweighting. CraEG is training-free, single-pass, and compatible with standard sampling strategies. Experiments on multiple models and benchmarks demonstrate improved generation performance, with gains in robustness and diversity metrics.
BibTeX
@misc{yang2026decoding,
  author = {Yixin Yang and Qingxiu Dong and Zhifang Sui},
  title = {{Decoding in Geometry: Alleviating Embedding-Space Crowding for Complex Reasoning}},
  year = {2026},
  eprint = {2601.22536},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.22536},
}
Figure