WakenLLM: Evaluating Reasoning Potential and Stability in LLMs via Fine-Grained Benchmarking

Zipeng Ling, Yuehao Tang, Shuliang Liu, Junqi Yang, Shenghong Fu, Chen Huang, Kejia Huang, Yao Wan, Zhichao Hou, Xuming Hu

Introduction

Large Language Models (LLMs) are increasingly fine-tuned for safety and "honesty," often resulting in models that are trained to abstain from answering when they are uncertain. While this reduces hallucinations, it has introduced a significant side effect: "Vague Perception." This phenomenon occurs when an LLM outputs "unknown" despite possessing the underlying knowledge and logical capacity to solve the problem. This paper introduces WAKENLLM, a pipeline designed to evaluate and "awaken" this latent reasoning potential, revealing that models are often far more capable than their initial responses suggest.

The Vague Perception Phenomenon

Vague Perception represents a mismatch between a model's potential ability and its inclination to abstain. The authors formalize this by identifying two primary roots of the problem:

  1. Reasoning Gap: Cases where the model fails to bridge logical steps initially but can successfully navigate the reasoning path when prompted or stimulated.
  2. Fact Understanding: Instances where the model fails to comprehend the context or facts provided in the input, leading to an incorrect assumption that the problem is unsolvable.
Conceptual overview of Vague Perception, illustrating the Reasoning Gap and Fact Understanding.

In addition to these, "Excessive Caution" plays a role, particularly in story-based tasks where models may be over-tuned to avoid making definitive claims.

Methodology: The WAKENLLM Pipeline

To address and quantify Vague Perception, the authors propose the WAKENLLM pipeline, a fine-grained, two-stage evaluation and mitigation framework.

The WAKENLLM pipeline architecture, detailing dataset preprocessing, stimulation stages, and evaluation methods.

The pipeline follows a structured process:

  • Sample Extraction: The system first identifies "Vague Perception" samples—cases where the model initially labels a solvable problem as "unknown."
  • Stage 1: Stimulation: The model is re-prompted using "Concise Stimulation" (a basic error notification) or "Detailed Stimulation" (step-by-step guidance). This stage measures the True Converting Rate (TCR)—the percentage of "unknown" samples successfully converted to correct answers.
  • Stage 2: Self-Reflection: For samples that remain incorrect after stimulation, the model is prompted to reflect on its own reasoning process. This helps identify if the model can self-correct when given a second chance.

The authors also introduce Acc(WAKENLLM), a new upper-bound accuracy metric that combines standard one-shot accuracy with the model's "potential" ability unlocked via the pipeline.

Experimental Setup and Datasets

The study evaluates six prominent LLMs, including GPT-4o, LLaMA-3.1 (8B, 70B, and 405B), and Qwen-2.5. The evaluation spans four diverse datasets, balanced 1:1 between solvable and unsolvable samples:

  • Fact-based: FLD (synthetic multi-step logic) and FOLIO (human-written first-order logic).
  • Story-based: ScienceQA (Physics and Biology) and Arts (Writing strategies and Figurative language).

Key Insights and Results

The experiments yielded several critical findings regarding the current state of LLM reasoning:

  • Massive Latent Potential: Without any weight updates or fine-tuning, LLMs achieved up to a 68.53% increase in accuracy on Vague Perception samples through the WAKENLLM pipeline. This suggests that a significant portion of "failed" reasoning is actually a failure of perception rather than a lack of capability.
  • The "Cautious Giant" Paradox: Surprisingly, larger models like GPT-4o and LLaMA-3.1-405B are more prone to Vague Perception than their smaller counterparts. Their extensive safety training makes them more likely to "give up" early on complex problems.
  • Conformity and Degradation: The study explored "Label Conformity," finding that while larger models are more resistant to being "gaslighted" by wrong labels, they are still susceptible. Additionally, the authors identified "Degradation," where a model moves from a wrong answer back to "unknown" during reflection, rather than finding the truth.
  • Baseline Gaps: Existing reasoning frameworks like Chain-of-Thought (CoT), Tree-of-Thought (ToT), and Plan-and-Solve only activate a small fraction of the latent potential identified by WAKENLLM, highlighting the need for more perception-aware approaches.

Conclusion and Future Directions

WAKENLLM demonstrates that current LLMs are significantly more capable than their one-shot performance suggests. The gap between standard accuracy and the Acc(WakenLLM) metric highlights a major opportunity for improving model utility. The authors suggest that future research should focus on perception-aware reasoning, designing models that better understand their own knowledge boundaries to reduce unnecessary abstention while maintaining safety.

Summary updated

Abstract

Large language models (LLMs) are increasingly trained to abstain on difficult questions by answering unknown. However, we observe that LLMs often misuse this option: they output unknown even when LLMs can actually solve the questions, or they fail to understand why questions are truly unsolvable. We formalize this mismatch between potential ability and the inclination of abstention as the Vague Perception phenomenon. We introduce the WakenLLM pipeline that (1) extracts Vague Perception samples and (2) measures how many of them can be converted to correct answers under stimulation. Based on stage-wise metrics (TCR, OCR, etc.) and the upper-bound accuracy Acc(WakenLLM), we quantify LLMs' reasoning potential beyond one-shot accuracy. Experiments on six LLMs suggest that, without further training or parameter revisions, LLMs can achieve up to a 68.53% increase in accuracy on Vague Perception samples through our designed pipeline. We further analyze how Vague Perception, Conformity and Degradation vary from model families and parameter sizes, and offer model selection strategies in multi-stage reasoning workflows. Finally, by comparing WakenLLM against mainstream reasoning baselines, both training and non-training ones, we show that existing baselines only activate a small portion of LLMs' reasoning potential, pointing to perception-aware reasoning as a promising direction for future LLM designing. Code and datasets are available at https://github.com/WakenLLMTeam/WakenLLM-toolkit.
BibTeX
@misc{ling2025wakenllm,
  author = {Zipeng Ling and Yuehao Tang and Shuliang Liu and Junqi Yang and Shenghong Fu and Chen Huang and Kejia Huang and Yao Wan and Zhichao Hou and Xuming Hu},
  title = {{WakenLLM: Evaluating Reasoning Potential and Stability in LLMs via Fine-Grained Benchmarking}},
  year = {2025},
  eprint = {2507.16199},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.16199},
}
Figure