Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space

Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang, Yelong Shen, Xin Eric Wang

TL;DR

"Soft Thinking" is a training-free method that allows Large Language Models (LLMs) to reason in a continuous concept space by using probability-weighted mixtures of token embeddings instead of discrete words. This approach improves reasoning accuracy by up to 2.48% while simultaneously reducing token usage by up to 22.4% across complex math and coding tasks.

What Problem Does This Paper Address?

Current LLMs typically reason using Chain-of-Thought (CoT), a process that forces the model to sample a single discrete token at every step. The authors argue that this creates a "discrete bottleneck." Human cognition is fluid and abstract; we don't always think in specific words before reaching a conclusion.

By committing to one specific token at each step, models can easily be led down suboptimal reasoning paths. Furthermore, generating long sequences of discrete text is computationally expensive. The challenge is to enable LLMs to explore multiple reasoning trajectories simultaneously in a more efficient, "soft" semantic space without requiring expensive retraining or architectural changes.

Methods

The authors introduce Soft Thinking, a training-free framework that operates in a continuous concept space.

  1. Concept Tokens: Instead of sampling a single token from the output probability distribution, the method treats the entire distribution as a "concept token."
  2. Continuous Embedding Injection: To move to the next step, the model calculates a weighted sum of all token embeddings based on the output probabilities. This "soft" embedding is then fed back into the model as the next input.
  3. Cold Stop Mechanism: Because LLMs are trained on discrete text, feeding them continuous embeddings puts them in an "out-of-distribution" (OOD) state, which can lead to model collapse (e.g., endless repetition). The authors implement "Cold Stop," which monitors the entropy of the output distribution. When the model becomes highly confident (low entropy) for several steps, the reasoning phase is terminated early to prevent collapse and save compute.
Figure 2: Soft Thinking replaces discrete tokens with soft, abstract concept tokens, enabling reasoning in continuous concept space.

Datasets

The method was evaluated on eight rigorous benchmarks across two domains:

  • Mathematics: Math500, AIME 2024, GSM8K, and GPQA-Diamond.
  • Programming: HumanEval, MBPP, and LiveCodeBench (using problems from Aug 2024 to Jan 2025 to ensure no data leakage).

Testing was performed on state-of-the-art open-source models including QwQ-32B, DeepSeek-R1-Distill-Qwen-32B, and DeepSeek-R1-Distill-Llama-70B.

Key Insights and Ideas

  • Implicit Parallelism: By using a mixture of embeddings, a single concept token can encapsulate multiple related meanings. This allows the model to implicitly explore various reasoning paths in "superposition" rather than being forced to pick one.
  • Theoretical Approximation: The authors provide a theoretical analysis showing that Soft Thinking serves as a first-order linearization of the full path-summation of all possible reasoning trajectories. Essentially, it approximates the average of many possible CoT chains in a single forward pass.
  • Efficiency through Fluidity: Soft Thinking often reaches the correct answer in fewer steps than standard CoT. By operating in a continuous space, the model can express complex intermediate ideas more concisely than natural language allows.
  • Interpretability Preserved: Despite the internal "softness," the most probable tokens at each step remain highly readable and logical, as shown in the authors' qualitative analysis of multiplication problems.
A heatmap showing how the probability distribution evolves; during exploratory phases, the distribution is more uniform, becoming "one-hot" during precise calculations.

Key Results

Soft Thinking consistently outperformed standard CoT across all models and tasks:

  • Accuracy Gains: The method improved pass@1 accuracy by up to 2.48 percentage points on average. Notably, on the difficult AIME 2024 math benchmark, QwQ-32B saw a jump of 6.45%.
  • Efficiency Gains: The reduction in generation length was significant, ranging from 11.6% to 22.4%. For the DeepSeek-R1-Distill-Qwen-32B model on math tasks, this meant a 22.4% reduction in the number of tokens needed to reach a correct solution.
  • Robustness: Unlike "Greedy CoT" (which is fast but often inaccurate), Soft Thinking maintains high performance while being faster than standard sampling-based CoT.
Figure 1: Soft Thinking vs. Chain-of-Thought thinking on mathematical and coding datasets. Soft Thinking consistently improves both accuracy (with improvements of up to 2.48% on pass@1 accuracy) and generation efficiency (achieving up to 22.4% reduction in generation length) across both tasks, without any training.

Limitations and Future Work

  • Out-of-Distribution (OOD) Issues: The primary limitation is that current LLMs are not trained on continuous concept tokens. This necessitates the "Cold Stop" heuristic to prevent the model from drifting into nonsense during long reasoning chains.
  • Training Integration: The authors suggest that future work should involve training or fine-tuning models specifically to handle these weighted mixtures, which could further stabilize the reasoning process and unlock even higher potential.
  • Heuristic Dependency: The performance of Soft Thinking depends on hyper-parameters like the entropy threshold for Cold Stop and the number of top tokens ($k$) used for the weighted sum.

Summary updated

Abstract

Human cognition typically involves thinking through abstract, fluid concepts rather than strictly using discrete linguistic tokens. Current reasoning models, however, are constrained to reasoning within the boundaries of human language, processing discrete token embeddings that represent fixed points in the semantic space. This discrete constraint restricts the expressive power and upper potential of such reasoning models, often causing incomplete exploration of reasoning paths, as standard Chain-of-Thought (CoT) methods rely on sampling one token per step. In this work, we introduce Soft Thinking, a training-free method that emulates human-like "soft" reasoning by generating soft, abstract concept tokens in a continuous concept space. These concept tokens are created by the probability-weighted mixture of token embeddings, which form the continuous concept space, enabling smooth transitions and richer representations that transcend traditional discrete boundaries. In essence, each generated concept token encapsulates multiple meanings from related discrete tokens, implicitly exploring various reasoning paths to converge effectively toward the correct answer. Empirical evaluations on diverse mathematical and coding benchmarks consistently demonstrate the effectiveness and efficiency of Soft Thinking, improving pass@1 accuracy by up to 2.48 points while simultaneously reducing token usage by up to 22.4% compared to standard CoT. Qualitative analysis further reveals that Soft Thinking outputs remain highly interpretable and readable, highlighting the potential of Soft Thinking to break the inherent bottleneck of discrete language-based reasoning. Code is available at https://github.com/eric-ai-lab/Soft-Thinking.
BibTeX
@inproceedings{zhang2025soft,
  author = {Zhen Zhang and Xuehai He and Weixiang Yan and Ao Shen and Chenyang Zhao and Shuohang Wang and Yelong Shen and Xin Eric Wang},
  title = {{Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=ByQdHPGKgU},
}
Figure