ThoughtProbe: Classifier-Guided LLM Thought Space Exploration via Probing Representations

Zijian Wang, Chang XuEMNLP 2025

TL;DR

This paper introduces ThoughtProbe, a novel framework that leverages the internal representations of Large Language Models (LLMs) to guide their reasoning process. By training a classifier to identify "thoughtful" reasoning steps within LLM activations, ThoughtProbe efficiently explores potential reasoning paths, leading to significant improvements in accuracy on mathematical and logical reasoning tasks.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown impressive capabilities in generating human-like text, but their reasoning abilities, especially for complex tasks like math word problems, can be inconsistent. While techniques like Chain-of-Thought (CoT) prompting have improved reasoning, they often require expensive fine-tuning or carefully crafted prompts. Researchers have observed that LLMs' internal "hidden representations" (activations within the neural network) correlate with their reasoning processes. However, directly manipulating these representations can disrupt the model's internal structure and degrade output quality. The challenge is to leverage these internal reasoning signals without negatively impacting the LLM's performance or requiring extensive retraining.

Methods

ThoughtProbe tackles this by treating LLM internal representations as discriminative signals rather than targets for manipulation. The core idea is to:

  1. Probe Representations: They train a simple classifier (Logistic Regression or SVM) on LLM hidden states to distinguish between "thoughtful" (CoT) and "non-thoughtful" reasoning steps. This classifier is trained on a dataset of question-response pairs labeled by GPT-4o.
  2. Classifier-Guided Exploration: This trained classifier is then used during inference to score potential reasoning steps. ThoughtProbe employs a tree-structured search (similar to Tree-of-Thoughts) where each node represents a reasoning step. At each step, the classifier assigns a "CoT score" to candidate continuations. The search prioritizes higher-scoring paths, effectively pruning less promising branches and focusing computational resources on more likely correct reasoning chains.
  3. Answer Aggregation: After exploring multiple reasoning paths, ThoughtProbe collects candidate answers from the leaf nodes of the search tree. Instead of simply picking the best one, it uses a "branch-aggregation" method to compute the value of each answer by summing the CoT scores of all supporting branches, leading to a more robust final answer selection.

Datasets

The paper uses several standard mathematical reasoning benchmarks for evaluation:

  • GSM8K: A challenging dataset of grade-school math problems.
  • MultiArith (MA): A dataset of multi-step arithmetic problems.
  • SVAMP: A dataset of math word problems.
  • MAWPS: A dataset of math word problems.
  • CoinFlips (CF): A logical reasoning benchmark.

The classifier itself is trained on a custom dataset derived from GSM8K, consisting of question-response pairs labeled as CoT or non-CoT using GPT-4o.

Key Insights and Ideas

  • Internal Representations as Signals, Not Targets: The key insight is that LLM internal states can be used as reliable signals to guide reasoning, rather than needing to be directly manipulated. This avoids the pitfalls of activation steering.
  • Simple Classifiers are Powerful: Surprisingly, simple classifiers like Logistic Regression can achieve high accuracy in distinguishing CoT from non-CoT reasoning within LLM activations, indicating that these reasoning patterns are encoded in a relatively accessible way.
  • Layer and Representation Type Matters: The effectiveness of probing representations varies across different layers and types of activations (e.g., attention vs. MLP). This suggests a nuanced understanding of how LLMs process reasoning is needed.
  • Classifier Logit as a Reliable Score: The logit output of the classifier is shown to be a theoretically sound and empirically effective score for ranking and guiding the search for promising reasoning paths.

Key Results

  • Significant Accuracy Gains: ThoughtProbe consistently outperforms various baselines (Greedy Decoding, Zero-shot CoT, Self-Consistency, etc.) across multiple LLMs (Mistral-7b, Gemma2-2b, Phi-1.5) and datasets. For example, on GSM8K, it achieves 38.18% accuracy with Mistral-7b, surpassing the strongest baseline by 14.01%.
  • Robustness Across LLM Scales: The method shows consistent improvements across different sizes of LLMs.
  • Effective Answer Selection: The proposed branch-aggregation method for answer selection is shown to be superior to simpler methods like majority voting or best-of-N.
  • Efficiency Trade-off: While more computationally intensive than single-path methods, ThoughtProbe offers a favorable balance between computational efficiency and reasoning capability compared to other tree-based methods.

Limitations and Future Work

  • Fixed Token Lengths: The current framework uses fixed token lengths to segment intermediate thoughts, which can sometimes disrupt natural reasoning. Future work could explore more flexible, semantic-aware splitting criteria.
  • Answer Quality vs. Coverage Gap: While the answer pool has high coverage of correct answers, the gap between coverage and accuracy suggests that current scoring and aggregation methods might not perfectly capture answer quality. Developing more sophisticated metrics is a future direction.
  • Inference Cost: ThoughtProbe's tree expansion and classifier evaluation increase inference time and latency compared to simpler methods. Future work aims to reduce this overhead through more efficient pruning or lightweight scoring.
  • Generalization of Classifier: While the classifier trained on GSM8K transfers well to other datasets, further research could explore even broader generalization capabilities.

Summary updated

BibTeX
@inproceedings{wang2025thoughtprobe,
  author = {Zijian Wang and Chang Xu},
  title = {{ThoughtProbe: Classifier-Guided LLM Thought Space Exploration via Probing Representations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure