TL;DR
PREGU (Partial Reasoning Guided by Uncertainty) is a novel framework that improves LLM reasoning by monitoring Shannon entropy during text generation. When the model becomes "uncertain" (high entropy), PREGU halts the process and triggers a targeted search in the latent space to refine the reasoning path, effectively mimicking the human "System 2" analytical thinking process.
What Problem Does This Paper Address?
Large Language Models (LLMs) are surprisingly good at reasoning, but they often stumble on complex, multi-step problems like advanced math or strategic planning. This is largely because they generate text "greedily"—one token at a time—without a built-in mechanism to simulate long-term outcomes or explore alternative paths. Once a model makes a slight logical error early in a chain, the mistake cascades.
Existing techniques like "Soft Reasoning" (SR) try to fix this by searching through the model's internal mathematical representations (latent space) rather than just the text. However, SR typically optimizes only from the very beginning of a prompt. This is computationally expensive and inefficient because it spends effort on parts of the sequence where the model might already be confident. The authors of this paper ask: Can we detect exactly when a model is confused and only apply intensive search at those specific "critical junctures"?
Methods
The authors introduce PREGU, a two-stage process that integrates uncertainty detection with latent-space refinement:
- Stage 1: Partial Reasoning (Token Space): The model begins generating multiple reasoning paths. At every step, PREGU calculates the Shannon Entropy of the output distribution. If the entropy exceeds a specific threshold ($\tau$), it signals that the model is "unsure" which token to pick next. The generation is immediately halted, creating a "partial reasoning prefix."
- Stage 2: Refinement (Latent Space): Instead of starting over, PREGU treats these partial paths as roots. It then uses the Soft Reasoning method to perform a localized search. This involves applying controlled Gaussian perturbations to the embedding of the first token and using Bayesian Optimization to find a version of that embedding that leads to the most coherent and correct final answer.
The quality of these refined paths is judged by a reward function that balances "correctness" (verified by the LLM itself) and "coherence" (syntactic fluency).
Datasets
The researchers evaluated PREGU using three popular open-source models (LLaMA-3-8B, Mistral-7B, and Qwen2-7B) across four challenging reasoning benchmarks:
- GSM8K: Grade-school math word problems.
- GSM-Hard: A more difficult version of GSM8K requiring deeper compositional inference.
- SVAMP: Math problems with superficial structural variations to test robustness.
- StrategyQA: Questions requiring multi-step commonsense reasoning.
Key Insights and Ideas
- Entropy as a "Metacognitive" Signal: The most striking finding is that entropy spikes aren't random. High entropy frequently occurs at logical transitions—words like "For", "Since", and "Given". These are the "forks in the road" where the model must choose a specific logical direction.
- System 1 vs. System 2: The authors frame PREGU as a computational version of Dual Process Theory. The fast, intuitive generation is "System 1," while the entropy-triggered latent search acts as the slow, analytical "System 2."
- Efficiency through Localization: By only triggering search when uncertainty is high, PREGU avoids wasting compute on "easy" parts of the reasoning chain, making the search more focused and effective than standard Soft Reasoning.
Key Results
PREGU consistently outperformed or matched the original Soft Reasoning method and other baselines like Chain-of-Thought (CoT) and Self-Consistency (SC).
- GSM-Hard Gains: On the toughest math benchmark, PREGU showed significant robustness. For example, with LLaMA-3-8B, PREGU achieved 35.2% accuracy, a notable jump from Soft Reasoning's 28.2%.
- Consistency: Across all models (LLaMA, Mistral, Qwen), PREGU showed higher accuracy on StrategyQA and GSM-Hard compared to standard prompting, suggesting that "stopping to think" is particularly beneficial for long-form logical chains.
- Linguistic Correlation: The paper includes a word cloud (Figure 2) showing that the highest entropy tokens are almost exclusively discourse connectives, proving that uncertainty is tied to the structural logic of the answer.
Limitations and Future Work
While promising, PREGU has a few hurdles to clear:
- The Threshold Problem: The entropy threshold ($\tau$) is currently a fixed hyperparameter. If it's set too high, the model never "stops to think"; too low, and the reasoning becomes fragmented. Future work aims to make this threshold dynamic based on the problem's complexity.
- Self-Verification Bias: The system relies on the LLM to verify its own answers during the search phase. If the model is fundamentally "blind" to a specific type of error, the reward function might lead it astray.
- Search Scope: Currently, the search only optimizes the very first token's embedding. The authors suggest that expanding this search "window" to multiple embeddings could lead to even smoother and more contextual refinements.
Summary updated