Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering

Anirudh Phukan, Shwetha Somasundaram, Apoorv Saxena, Koustava Goswami, Balaji Vasan Srinivasan

Introduction

Large Language Models (LLMs) have revolutionized how we interact with information, particularly in contextual question answering (QA) where a model generates a response based on a provided document. However, a persistent challenge remains: trust. While LLMs often weave together their own "glue text" with factual spans copied directly from the source, they rarely provide explicit citations. This lack of attribution makes it difficult for users to verify the authenticity of a response, leading to concerns about hallucinations and reliability.

Researchers from Adobe have recently introduced a novel, training-free method that addresses this gap by "peering" into the internal hidden states of LLMs. Their approach demonstrates that models possess an inherent, latent awareness of when they are copying information, allowing for precise, token-level attribution without the need for expensive fine-tuning or secondary processing passes.

The Attribution Gap

Existing solutions for source attribution generally fall into three categories, each with significant drawbacks:

  1. Fine-tuning: Training models specifically to cite sources is computationally expensive and requires constant retraining as new models or data emerge.
  2. Retrieve-then-read: Using external retrieval models adds significant computational overhead and complexity to the system architecture.
  3. Post-generation attribution: Asking the model (or a separate evaluator like GPT-4) to find sources after the response is generated is often imprecise, slow, and prone to its own set of hallucinations.

The authors hypothesize that these external methods are unnecessary because the LLM already "knows" when it is copying text. This information is encoded within the model's internal representations during the initial generation process.

A Training-Free Approach to Attribution

The proposed method leverages the hidden layer representations (embeddings) of tokens during a single forward pass. It operates through two primary sub-tasks:

Sub-task 1: Token Attribution Identification

The system first identifies which tokens in the generated answer were copied from the source document. By calculating the cosine similarity between the hidden state of an answer token and all tokens in the source document, the system can determine if a token is "extractive." If the similarity exceeds a specific threshold ($\theta$), the token is marked as copied.

Sub-task 2: Corresponding Token Mapping

Once copied tokens are identified, the system maps them back to their exact location in the source. It identifies "anchor tokens"—those with the highest similarity scores—and employs a sliding window approach. This allows the system to find the specific span in the document that most closely matches the generated span's average embedding, ensuring high-granularity mapping.

Because this method is training-free, it is model-agnostic and can be applied to various architectures, including Llama, Mistral, Yi, and others, simply by accessing their existing layers.

Benchmarking with VERI-GRAN

To evaluate the effectiveness of this approach, the researchers utilized the existing QuoteSum dataset and introduced a new benchmark: VERIFIABILITY-GRANULAR (VERI-GRAN).

Unlike previous datasets that focused on sentence-level or response-level attribution, VERI-GRAN provides token-level annotations for LLM-generated responses. This allows for a much finer evaluation of attribution accuracy, reflecting the nuanced way LLMs actually construct answers from source text.

Insights into Model Behavior

The research revealed several key insights into the internal mechanics of Transformers:

  • Internal Awareness: The most significant finding is that LLMs have a latent awareness of their own copying behavior. This awareness is captured in the embeddings, suggesting that attribution is a fundamental property of how these models process context.
  • Layer-Wise Specialization: The study found that different layers of the model excel at different aspects of attribution. Early layers are generally better at identifying that a token was copied (low-level feature matching). In contrast, middle and later layers are superior at disambiguation—determining which specific instance of a word was copied when that word appears multiple times in the source.
  • Model Agnosticism: This "copying awareness" was consistent across various model families (Llama-7b/70b, Mistral-7b, Yi-6b, OPT-350m), indicating it is a universal characteristic of Transformer-based LLMs.

Results and Performance

The proposed method demonstrated remarkable accuracy and efficiency:

  • Outperforming GPT-4: On the VERI-GRAN dataset, the method outperformed GPT-4 by approximately 15% in paragraph-level attribution accuracy.
  • High Precision: On the QuoteSum dataset, the method achieved a paragraph-level attribution accuracy of ~90%, matching GPT-4's performance while providing much higher granularity (token-level vs. sentence-level).
  • Efficiency: By utilizing hidden states from the initial generation pass, the method avoids the computational costs and hallucination risks associated with prompting a model to cite its own work post-hoc.

Limitations and Future Directions

While the method is highly effective for verbatim (word-for-word) copied spans, it currently does not handle paraphrased information where the model captures the intent but alters the wording. Additionally, the approach is constrained by the LLM's maximum context window (typically 2048 to 4096 tokens).

Future work will focus on expanding the approach to handle paraphrased attributions and exploring its utility in cross-lingual information extraction, such as mapping a response in one language back to a source document in another.

Summary updated

BibTeX
@inproceedings{phukan2024peering,
  author = {Anirudh Phukan and Shwetha Somasundaram and Apoorv Saxena and Koustava Goswami and Balaji Vasan Srinivasan},
  title = {{Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.682},
}
Figure