When Jason Wei et al. introduced Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, it fundamentally changed how we interact with large language models (LLMs). By generating intermediate reasoning steps, models could suddenly solve complex mathematical, symbolic, and logical problems that previously stumped them.
However, this emergent intelligence comes with a severe operational penalty: inference-time latency. Because standard LLMs generate text autoregressively—one token at a time—a reasoning chain that spans hundreds of tokens requires hundreds of sequential forward passes through the network. This sequential bottleneck introduces massive computational overhead and high costs, making high-performance reasoning models difficult to deploy in real-time applications, agentic workflows, or high-throughput production environments.
Why Token-Space Reasoning is Inefficient
To understand why token-by-token reasoning is inefficient, we must look at what happens inside the model's hidden layers. In standard Chain-of-Thought (CoT), the model uses natural language tokens as its working memory. It must map its internal abstract concepts down to discrete words, emit them, and then re-encode them in the next step.
Intriguingly, research by Anum Afzal et al. in Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion demonstrates that an LLM's internal hidden states often encode whether a reasoning path will succeed at "time step zero"—before the model has even typed out its explanation. This implies that generating extensive natural language sequences is often a redundant, superficial overhead rather than a strict computational necessity for the underlying logic. The challenge, therefore, is to decouple the model's internal deliberation from its external verbalization.
Thinking Silently in Latent Space
One of the most promising directions for eliminating this latency is latent reasoning—forcing the model to "think" using continuous vectors rather than discrete text tokens.
For instance, Wenhui Tan et al. propose Compressed Latent Reasoning (CoLaR) in Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains. Instead of generating a wordy sentence step-by-step, the model predicts compressed embeddings that encapsulate multiple tokens or concepts simultaneously. This reduces the reasoning chain length by over 80% while maintaining accuracy.
Similarly, Jiayu Liu et al. introduce MARCOS in MARCOS: Deep Thinking by Markov Chain of Continuous Thoughts, which models reasoning as a hidden Markov chain of high-dimensional continuous vectors. By operating entirely within the latent space, the framework achieves up to a 15.7× inference speedup.
Other approaches, such as SemCoT by Yinhan He et al. (SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokens) and iCLP by Di Niu et al. (iCLP: Large Language Model Reasoning with Implicit Cognition Latent Planning), inject a small number of "implicit" hidden tokens or discrete latent codes to guide the model's internal planning. This bypasses the need for verbose meta-strategy verbalization, allowing the model to compute complex transitions in a single forward pass rather than across dozens of text tokens.
Incentivizing Intelligence per Token
Another approach focuses on pruning the unnecessary verbosity that models naturally adopt during reinforcement learning (RL). When models are rewarded purely for correctness, they often become excessively wordy, repeating variations of the same logical steps or generating redundant rationales.
To counteract this, Xin Dong et al. develop DLER in DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning. By fixing optimization issues in standard length penalties, they reduce output lengths by over 70% without losing accuracy.
In a similar vein, Razvan-Gabriel Dumitru et al. introduce ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models, which uses an LLM-as-a-judge to provide a "conciseness reward." This trains the model to maximize its semantic compactness, achieving up to a 31× reduction in token usage on math problems. By shifting the optimization landscape, these methods ensure that every generated token carries maximum cognitive weight.
Speculative Reasoning and Hierarchical Parallelism
When text generation cannot be avoided, researchers leverage speculative execution to break the sequential bottleneck. Traditional speculative decoding matches exact tokens, which fails on fluid reasoning chains where multiple phrasing choices are valid.
To solve this, Rui Pan et al. design SpecReason in SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning. SpecReason allows a small, fast model to speculate entire intermediate reasoning steps, while the large base model acts as a "semantic critic" to verify the underlying logic rather than the exact wording.
This concept is extended in Scaling Speculative Decoding with Lookahead Reasoning, which introduces a hierarchical layer of parallelism. By validating entire reasoning blocks based on semantic equivalence rather than token-level matches, these systems achieve up to a 3.0× speedup, proving that verification is computationally cheaper than generation.
Internalizing and Distilling the Reasoning Trace
Can we eliminate the reasoning chain entirely during inference? Several researchers focus on "internalizing" System 2 (slow, deliberate) thinking into System 1 (fast, intuitive) execution via distillation.
Yongxin Deng et al. introduce the CogniDual Framework: Self-Training Large Language Models within a Dual-System Theoretical Framework for Improving Cognitive Tasks, which trains models to internalize their own CoT outputs, achieving high logical accuracy without the computational delay of step-by-step generation.
For smaller models, Xiaoshu Chen proposes an Adaptive Thinking Mechanism in Post-Semantic-Thinking: A Robust Strategy to Distill Reasoning Capacity from Large Language Models. This allows a small model to dynamically choose between "thinking before answering" and "answering before thinking," mitigating redundant overhead for straightforward tasks and yielding up to a 50× speedup.
Furthermore, Jinghan Jia et al. demonstrate in EPiC: Towards Lossless Speedup for Reasoning Training through Edge-Preserving CoT Condensation that we can prune the middle of long CoT traces during distillation. Keeping only the initial problem framing and final solution convergence allows for a 34% faster training pipeline without sacrificing the distilled model's ultimate accuracy.