Abstract
Large Language Models (LLMs) frequently exhibit "greedy reasoning," producing logically structured but factually inconsistent chains of thought. This paper introduces the QK-score, a mechanistic metric derived from the internal attention mechanisms of transformers. By measuring the dot-product alignment between specific query and key vectors in specialized "reasoning heads," we can quantify the logical consistency of a model's internal state. Our findings demonstrate that the QK-score serves as a reliable "logical thermometer," often outperforming the model's final output probabilities in distinguishing valid from invalid inferences across various logical benchmarks and model scales.
Introduction
While Large Language Models have demonstrated impressive capabilities in multi-step reasoning through Chain-of-Thought (CoT) prompting, their internal consistency remains a significant challenge. Models often generate sequences that appear logically sound but contain subtle hallucinations or invalid transitions. Current methods for evaluating these transitions, such as ablation studies, are computationally prohibitive for large-scale models.
This research addresses the need for a lightweight, internal mechanism to assess logical coherence. We propose that the transformer's attention mechanism—specifically the alignment between queries and keys—contains latent signals that indicate the validity of a logical step before the final output is even generated.
Methodology: The QK-Score
The core of our approach is the QK-score, a metric that leverages the fundamental mechanics of the transformer architecture. In a standard attention head, the relationship between tokens is determined by the dot product of a Query ($Q$) and a Key ($K$).
The Extraction Process
- Forward Pass: The model processes a prompt containing a logical context, a specific statement, and a candidate answer (e.g., "true" or "false").
- Activation Extraction: Rather than relying on the final softmax output, we extract the raw $q^\top k$ values from all attention heads during the forward pass.
- Head Selection: Using a small calibration set (~400 samples), we identify "best heads"—specific attention heads that demonstrate high accuracy in separating valid logical inferences from invalid ones.
- Scoring: The QK-score from these designated heads is used as the primary metric for truthfulness.
Unlike standard attention weights, the QK-score is independent of positional encodings, allowing it to focus strictly on the semantic alignment between a premise and its conclusion.
Experimental Evaluation
We evaluated the QK-score across three diverse logical benchmarks to test its robustness and generalizability:
- PrOntoQA-OOD: A synthetic first-order logic dataset tested in Out-of-Distribution settings, including varying reasoning depths and the inclusion of "distractors" (irrelevant facts).
- PARARULE Plus: A dataset requiring fixed-depth deductive reasoning for true/false questions.
- Extended-Multi-LogiEval: An expanded benchmark featuring a balanced distribution of logical schemes such as Modus Ponens and Modus Tollens.
Our experiments spanned multiple model architectures, including Qwen2.5 (1.5B) and LLaMA 3 (70B), to ensure the phenomenon was not architecture-specific.
Key Findings
1. Superior Predictive Accuracy
On the PARARULE Plus dataset, the QK-score method outperformed the standard baseline (the model's highest probability output) by over 10% in several configurations. This suggests that the "correct" logical signal is often present within the model's internal layers but may be obscured by subsequent processing or decoding biases.
2. Resilience to Complexity and Noise
As reasoning depth increased (from 1 to 5 hops), the performance gap between the QK-score and the baseline widened. Furthermore, while distractors typically degrade the accuracy of an LLM's final answer, the QK-scores in specialized reasoning heads remained remarkably stable, indicating that these heads effectively filter for semantic relevance.
3. Localization of Logic
Our research suggests that logical reasoning is not a diffuse, whole-model phenomenon. Instead, it is localized within specific attention heads that act as "verification anchors." These heads maintain consistent performance even as the reasoning chain grows more complex.
Discussion and Limitations
While the QK-score provides a powerful tool for interpretability, it requires a calibration phase to identify the relevant reasoning heads for a specific model. Additionally, our current study focused primarily on deductive logic; further research is required to determine if QK-alignment holds for inductive or creative reasoning tasks.
Future work may involve integrating QK-scores directly into the decoding process, potentially "steering" models toward more logically consistent outputs in real-time. By utilizing these internal "logical thermometers," we can move toward LLMs that are not only more accurate but also more internally coherent.
Summary updated