Abstract
Large Language Models (LLMs) integrate Parametric Knowledge (PK) from internal weights and Contextual Knowledge (CK) from prompts to generate responses. Traditional mechanistic interpretability models this dynamic as a 1D "tug-of-war," which fails when both sources are active simultaneously. We introduce a rank-2 subspace disentanglement method to independently track PK and CK contributions token-by-token. Our findings reveal that hallucinations are driven by PK overrides, while Chain-of-Thought (CoT) prompting enhances grounding by aligning model representations with the CK axis.
The Challenge of Knowledge Interaction
When an LLM generates a Natural Language Explanation (NLE), it balances information memorized during training (Parametric Knowledge) with information provided in the prompt (Contextual Knowledge). Understanding this balance is critical for mitigating hallucinations and improving context grounding.
Historically, this interaction was modeled as a binary conflict on a rank-1 subspace (a 1D line). However, this zero-sum approach is insufficient for scenarios where PK and CK are complementary or supportive. Furthermore, past analyses often focused only on the final answer token, missing the complex dynamics that evolve throughout a multi-step explanation.
Disentangling Knowledge via Rank-2 Subspaces
To address these limitations, we propose a rank-2 projection subspace—a 2D plane that can independently represent PK and CK directions.
The methodology involves:
- Activation Patching: Identifying mid-to-late layers (e.g., layers 13–18 in Llama-3.1-8B) that encode the intent to rely on specific knowledge sources.
- Subspace Learning: Defining a 2D plane using two orthogonal vectors for the PK and CK directions.
- Token-by-Token Tracking: Mathematically disentangling the contribution of each source at every generated token, allowing for a granular view of the model's internal state during reasoning.
Experimental Framework
We evaluated three open-weight models—Llama-3.1-8B-Instruct, Gemma-2-9B-it, and Mistral-v0.3 7B—across diverse datasets. These included reasoning tasks like StrategyQA and OpenBookQA, as well as hallucination-focused datasets like RAGTruth and Dollo (AC). We specifically looked at scenarios where PK and CK were in conflict, supportive, or irrelevant to each other.
Insights into Hallucinations and Reasoning
Our analysis yielded several key insights:
- Non-Zero-Sum Dynamics: In supportive contexts, models do not choose one source over the other; instead, they lean heavily on PK while using CK as a regularizer. A rank-1 model fails to identify these simultaneous activations.
- Hallucinations as Parametric Overrides: Internal geometry reveals that hallucinated text spans align strongly with the PK axis. This suggests hallucinations are systematic biases where the model defaults to internal weights over provided context.
- Mechanics of Chain-of-Thought (CoT): CoT prompting shifts the model's internal state into a low-rank subspace that aligns more closely with the CK direction, explaining its superior grounding.
- Temporal Fluctuations: Tracking explanations token-by-token shows that models often start grounded in CK but "wobble" toward PK as uncertainty increases in longer sequences.
Quantitative Results
The necessity of the rank-2 approach is underscored by the Explained Variance Ratio (EVr). While rank-1 subspaces fail to capture the variance in complex interactions, the rank-2 subspace consistently reaches near-total cumulative explained variance.
The learned subspaces are remarkably stable, maintaining high stability scores (~0.98) across adjacent layers and high cosine similarity even when prompts are paraphrased.
Conclusion and Future Directions
While this methodology currently requires access to internal activations—limiting its use to open-weight models—it provides a powerful diagnostic tool. Future work may leverage these rank-2 signals at inference time to actively steer models, forcing contextual grounding and preemptively suppressing hallucinations.
Summary updated