LargePiG: Your Large Language Model is Secretly a Pointer Generator

Zhongxiang Sun, Zihua Si, Xiaoxue Zang, Kai Zheng, Yang Song, Xiao Zhang, Jun XuarXiv 2024 · OpenReview

TL;DR

LargePiG is a training-free, plug-and-play decoding method that transforms Large Language Models (LLMs) into Pointer-Generator networks to mitigate hallucinations during query generation. By leveraging internal attention weights to copy factual tokens directly from the input and using early-exit layer differences to decide when to copy, it significantly improves query relevance and factuality with minimal computational overhead.

What Problem Does This Paper Address?

Query generation—the process of automatically creating search queries from documents or videos—is essential for modern search engines and short-video recommendation platforms. However, when LLMs are tasked with generating these queries, they frequently suffer from two distinct types of hallucinations:

  1. Factuality Hallucinations: Generating queries containing false facts or details that contradict the source material.
  2. Relevance Hallucinations: Generating queries that are factually true in a general sense but completely irrelevant to the specific input context.

Traditional methods to fix these issues rely on expensive model retraining or complex post-processing retrieval steps. This paper addresses the challenge at the decoding source by decoupling content (factual tokens extracted directly from the input) from form (the syntactic structure generated by the LLM).

Methods

The authors introduce LargePiG (Large Pointer-Generator), a model-agnostic, training-free plug-in that forces an LLM to act as a pointer-generator network using purely internal, pre-trained mechanisms:

  • Pointer Attention Distribution: Instead of training a separate pointer network to locate source words, LargePiG extracts the LLM's internal self-attention weights from the last generated token back to the source tokens.
  • Vocabulary Distribution: It retains the standard next-token probability distribution from the final layer to maintain linguistic fluency and grammatical structure.
  • Copy Probability ($p_{copy}$): To dynamically decide whether to copy a word from the text or generate a new one, LargePiG compares the vocabulary distributions of the model's high layers (via early exiting) against the final anchor layer.
Figure 1: The architecture of the proposed plug-in and training-free method LargePiG. Pointer Attention Distribution (§ 2.1) from the LLM’s self-attention weights, Vocabulary Distribution (§ 2.2) from the output of the original LLM, Copy Probability (§ 2.3) from the difference between the vocabulary distribution of the model’s high layers and the last layer.

Datasets

To rigorously evaluate their approach, the authors introduced two new benchmarks specifically tailored for truthful query generation:

  • TruthfulVQG: A Chinese dataset collected from a real short-video platform containing video titles, dialogue (ASR), and on-screen text (OCR).
  • TruthfulDQG: An English document-based dataset adapted from the MS-MARCO machine reading comprehension dataset.
  • TruthfulVQG-M: A multimodal variant used to test Large Vision-Language Models (LVLMs) using video covers.

They also validated copy mechanics using the standard SQuAD dataset and general factuality via the FACTOR benchmark.

Key Insights and Ideas

  • Attention is Truthful, FFNs Hallucinate: The authors note that factual knowledge is primarily stored within the Feed-Forward Network (FFN/MLP) modules of an LLM. These modules can override the input context with incorrect pre-trained biases. Conversely, self-attention modules remain highly focused on the provided context, making them excellent "pointers."
  • Linguistic Complexity Signals Copying: A key discovery is that LLM predictions for simple function words (prepositions, conjunctions, auxiliary verbs) stabilize early in the higher layers of the network. In contrast, complex factual words (names, places, dates) continue to evolve until the very last layer. LargePiG exploits this distributional delta to dynamically calculate the copy probability without adding any extra parameters.

Key Results

  • Across-the-Board Improvements: LargePiG consistently outperformed base decoding and state-of-the-art contrastive decoding methods (like DoLa) across multiple models (Qwen1.5-7B and LLaMA2-7B) on both benchmarks.
Table 1: Performance comparisons between LargePiG and the baselines. The boldface represents the best performance. ‘†’ means improvements are significant (paired t-test at p-value < 0.05).
  • Multimodal Success: When applied to LLaVA-7B and LLaVA-13B, LargePiG successfully boosted truthfulness metrics (MC1, MC2, MC3) by mapping image tokens to text tokens in the embedding space.
Table 2: Experimental results on multimodal data.
  • Enhanced Copying on Long Inputs: On the SQuAD dataset, LargePiG significantly improved F1 scores, with the most pronounced gains occurring on long context inputs where traditional LLMs typically struggle with copy fidelity.
Figure 2: Left: Semantic similarity win rate of Qwen1.5-7B-Chat with LargePiG vs without LargePiG on TruthfulVQG. Right: Performance of Qwen1.5-7B-Chat vs Qwen1.5-7B-Chat + LargePiG on SQuAD.
  • Negligible Latency: The method introduces a maximum decoding latency increase of only 6% to 10%, proving its practical viability for production environments.

Limitations and Future Work

  • Scale Constraints: Due to limited computing resources, empirical validation was primarily restricted to 7B-parameter models. Verifying the effects of LargePiG on larger scales (e.g., 70B+ models) remains future work.
  • Layer Selection Optimization: Currently, LargePiG relies on a heuristic selection of the last layer's attention weights. The authors suggest that exploring supervised probing techniques could optimize layer selection for even better pointer distributions.
  • RAG Integration: A promising future direction is directly applying LargePiG to Retrieval-Augmented Generation (RAG) pipelines to suppress downstream hallucinations when synthesizing retrieved context.

Summary updated

Abstract

Recent research on query generation has focused on using Large Language Models (LLMs), which despite bringing state-of-the-art performance, also introduce issues with hallucinations in the generated queries. In this work, we introduce relevance hallucination and factuality hallucination as a new typology for hallucination problems brought by query generation based on LLMs. We propose an effective way to separate content from form in LLM-generated queries, which preserves the factual knowledge extracted and integrated from the inputs and compiles the syntactic structure, including function words, using the powerful linguistic capabilities of the LLM. Specifically, we introduce a model-agnostic and training-free method that turns the Large Language Model into a Pointer-Generator (LargePiG), where the pointer attention distribution leverages the LLM's inherent attention weights, and the copy probability is derived from the difference between the vocabulary distribution of the model's high layers and the last layer. To validate the effectiveness of LargePiG, we constructed two datasets for assessing the hallucination problems in query generation, covering both document and video scenarios. Empirical studies on various LLMs demonstrated the superiority of LargePiG on both datasets. Additional experiments also verified that LargePiG could reduce hallucination in large vision language models and improve the accuracy of document-based question-answering and factuality evaluation tasks.
BibTeX
@misc{sun2024largepig,
  author = {Zhongxiang Sun and Zihua Si and Xiaoxue Zang and Kai Zheng and Yang Song and Xiao Zhang and Jun Xu},
  title = {{LargePiG: Your Large Language Model is Secretly a Pointer Generator}},
  year = {2024},
  eprint = {2410.11366},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.11366},
}
Figure