Abstract
Large Language Models (LLMs) often exhibit poor calibration in their external outputs, making it difficult to distinguish correct reasoning from confident hallucinations. This paper explores internal neuron activation patterns as a more reliable signal of correctness. We introduce Neuron Agreement Decoding (NAD), a framework that identifies correct responses by monitoring internal "consensus" and activation sparsity. Our results show that NAD can identify correct trajectories with 99% fewer tokens than standard ensemble methods, particularly in complex math and coding tasks.
Introduction: The Calibration Gap
To optimize LLM performance, researchers typically use "Self-Consistency" methods, sampling multiple responses and selecting the majority answer. However, this approach is computationally expensive—requiring full generation of all candidates—and fails for open-ended tasks like code generation where multiple correct answers may not share the same text.
Existing confidence metrics, such as token probabilities or self-evaluation, are often "low-resolution" projections of the model's state. As shown in Figure 1, there is a fundamental disconnect between external behaviors (text output) and internal behaviors (activation patterns). This research investigates whether looking "under the hood" at neuron activations provides a clearer signal of truth.
Methodology: Neuron Agreement Decoding (NAD)
NAD monitors which neurons fire during the early stages of generation to evaluate the "health" of a reasoning path. The framework employs three primary selection strategies:
- kNN-Agreement: Measures similarity in activation patterns to find the most "in-sync" response.
- Global Medoid: Identifies the central response in activation space, representing the model's internal consensus.
- MinAct (Minimal Activation): Selects the response with the most sparse activation pattern, based on the observation that correct paths are often more efficient.
A key innovation is the Early Stopping Strategy, which allows NAD to prune unpromising trajectories within the first 32 tokens, drastically reducing computational overhead.
Key Insights: Sparsity and Consensus
The researchers identified two critical phenomena in the model's internal dynamics:
1. The Sparsity Signal
Correct responses tend to activate significantly fewer unique neurons than incorrect ones. When a model "knows" the correct path, its internal state remains focused and sparse. Conversely, incorrect reasoning often leads to "over-exploration," where the model activates a chaotic, dense set of neurons as it struggles to resolve the prompt.
2. Internal Consensus
Even when final text outputs differ, correct reasoning paths tend to form tight clusters in the high-dimensional neuron activation space. Incorrect paths diverge as outliers. Figure 5 illustrates these dynamics, showing how correct answers (green) cluster together and maintain lower unique neuron counts compared to incorrect ones (red).
Results and Performance
NAD was evaluated on challenging benchmarks including AIME24 (math) and HumanEval (code) using Qwen3 and DeepSeek-R1 models.
- Efficiency: NAD achieved a 99% reduction in token usage via early stopping with negligible accuracy loss.
- Coding Tasks: Unlike majority voting, NAD excels at open-ended coding tasks by identifying the underlying logic rather than exact text matches.
- Early Prediction: The system reliably predicts success within the first 32 tokens, enabling aggressive pruning of "noisy" internal trajectories.
Limitations and Future Directions
While NAD is highly efficient in terms of compute, storing high-dimensional activation vectors introduces memory overhead. Future work may involve using bitset or bitmap encodings to compress these signals. Additionally, while the "MinAct" signal is robust in mathematics, its strength varies in more open-ended domains like software engineering, suggesting a need for task-specific selector optimization.
Summary updated