Multi-Step Knowledge Interaction Analysis via Rank-2 Subspace Disentanglement

Sekh Mainul Islam, Pepa Atanasova, Isabelle AugensteinarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) integrate Parametric Knowledge (PK) from internal weights and Contextual Knowledge (CK) from prompts to generate responses. Traditional mechanistic interpretability models this dynamic as a 1D "tug-of-war," which fails when both sources are active simultaneously. We introduce a rank-2 subspace disentanglement method to independently track PK and CK contributions token-by-token. Our findings reveal that hallucinations are driven by PK overrides, while Chain-of-Thought (CoT) prompting enhances grounding by aligning model representations with the CK axis.

The Challenge of Knowledge Interaction

When an LLM generates a Natural Language Explanation (NLE), it balances information memorized during training (Parametric Knowledge) with information provided in the prompt (Contextual Knowledge). Understanding this balance is critical for mitigating hallucinations and improving context grounding.

Historically, this interaction was modeled as a binary conflict on a rank-1 subspace (a 1D line). However, this zero-sum approach is insufficient for scenarios where PK and CK are complementary or supportive. Furthermore, past analyses often focused only on the final answer token, missing the complex dynamics that evolve throughout a multi-step explanation.

Disentangling Knowledge via Rank-2 Subspaces

To address these limitations, we propose a rank-2 projection subspace—a 2D plane that can independently represent PK and CK directions.

Overview of the rank-2 subspace disentanglement pipeline, showing the transition from 1D to 2D modeling.

The methodology involves:

  1. Activation Patching: Identifying mid-to-late layers (e.g., layers 13–18 in Llama-3.1-8B) that encode the intent to rely on specific knowledge sources.
  2. Subspace Learning: Defining a 2D plane using two orthogonal vectors for the PK and CK directions.
  3. Token-by-Token Tracking: Mathematically disentangling the contribution of each source at every generated token, allowing for a granular view of the model's internal state during reasoning.

Experimental Framework

We evaluated three open-weight models—Llama-3.1-8B-Instruct, Gemma-2-9B-it, and Mistral-v0.3 7B—across diverse datasets. These included reasoning tasks like StrategyQA and OpenBookQA, as well as hallucination-focused datasets like RAGTruth and Dollo (AC). We specifically looked at scenarios where PK and CK were in conflict, supportive, or irrelevant to each other.

Insights into Hallucinations and Reasoning

Our analysis yielded several key insights:

  • Non-Zero-Sum Dynamics: In supportive contexts, models do not choose one source over the other; instead, they lean heavily on PK while using CK as a regularizer. A rank-1 model fails to identify these simultaneous activations.
  • Hallucinations as Parametric Overrides: Internal geometry reveals that hallucinated text spans align strongly with the PK axis. This suggests hallucinations are systematic biases where the model defaults to internal weights over provided context.
  • Mechanics of Chain-of-Thought (CoT): CoT prompting shifts the model's internal state into a low-rank subspace that aligns more closely with the CK direction, explaining its superior grounding.
  • Temporal Fluctuations: Tracking explanations token-by-token shows that models often start grounded in CK but "wobble" toward PK as uncertainty increases in longer sequences.

Quantitative Results

The necessity of the rank-2 approach is underscored by the Explained Variance Ratio (EVr). While rank-1 subspaces fail to capture the variance in complex interactions, the rank-2 subspace consistently reaches near-total cumulative explained variance.

Cumulative Explained Variance Ratio (EVr) across different models and datasets, demonstrating that rank-2 is sufficient to capture knowledge interactions.

The learned subspaces are remarkably stable, maintaining high stability scores (~0.98) across adjacent layers and high cosine similarity even when prompts are paraphrased.

Conclusion and Future Directions

While this methodology currently requires access to internal activations—limiting its use to open-weight models—it provides a powerful diagnostic tool. Future work may leverage these rank-2 signals at inference time to actively steer models, forcing contextual grounding and preemptively suppressing hallucinations.

Summary updated

Abstract

Natural Language Explanations (NLEs) describe how Large Language Models (LLMs) make decisions by drawing on external Context Knowledge (CK) and Parametric Knowledge (PK). Understanding the interaction between these sources is key to assessing NLE grounding, yet these dynamics remain underexplored. Prior work has largely focused on i) single-step generation and ii) modeled PK--CK interaction as a binary choice within a rank-1 subspace. This approach overlooks richer interactions and how they unfold over longer generations, such as complementary or supportive knowledge. We propose a novel rank-2 projection subspace that disentangles PK and CK contributions more accurately and use it for the first multi-step analysis of knowledge interactions across longer NLE sequences. Experiments across four QA datasets and three open-weight LLMs demonstrate that rank-1 subspaces struggle to represent diverse interactions, whereas our rank-2 formulation captures them effectively, highlighting PK alignment for supportive interactions and CK alignment for conflicting ones. Our multi-step analysis reveals that hallucinated generations exhibit strong alignment with the PK direction, though context-faithful generations maintain a more balanced alignment between PK and CK.
BibTeX
@misc{islam2025multi,
  author = {Sekh Mainul Islam and Pepa Atanasova and Isabelle Augenstein},
  title = {{Multi-Step Knowledge Interaction Analysis via Rank-2 Subspace Disentanglement}},
  year = {2025},
  eprint = {2511.01706},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.01706},
}
Figure