TL;DR
This paper introduces 𝒜³ (Automatic Alignment Framework), a novel method for automatically generating high-quality training data for attributed text generation. By addressing the scarcity of human-annotated data, 𝒜³ enables LLMs to produce more reliable and verifiable responses with citations, significantly outperforming existing methods.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly used for information seeking, but their outputs can be unreliable and lack verifiable sources. Attributed text generation aims to solve this by providing citations for each claim, enhancing trustworthiness. However, creating high-quality training data for this task is challenging due to the high cost of human annotation and the difficulty of aligning LLMs to generate accurate, cited responses without focusing on irrelevant information. Existing approaches often struggle with noisy retrieved documents and the inherent limitations of LLMs in following complex instructions for citation.
Methods
The 𝒜³ framework tackles these challenges through a two-stage process: Data Construction and Model Training.
Data Construction: * Entity Matching Based Document Clustering: This strategy groups related documents to ensure coherence and facilitate multi-source citations. * Q&A Pair Extraction & Noisy Document Augmentation: LLMs are used to generate attributed question-answer pairs (Qi, Ri, Ci) from the clustered documents. A "noisy document augmentation" strategy is employed to simulate real-world scenarios where retrieved passages might contain irrelevant information, making the training more robust. * Automatic Dispreferred Response Construction: For preference optimization, the framework automatically generates "dispreferred" responses by altering the preferred ones. This involves adding/removing/changing citations or emphasizing irrelevant documents, creating challenging examples for the LLM to learn from.
Model Training: The generated data is used for two main training stages:
- Supervised Fine-Tuning (SFT): Standard fine-tuning using the generated attributed Q&A pairs.
- Preference Optimization (PO): Using Direct Preference Optimization (DPO) to train the model on preferred and dispreferred response pairs, further refining its ability to generate accurate and well-cited answers.
Datasets
The paper utilizes existing corpora and introduces a new dataset generation methodology.
- Wikipedia: Used as a primary data source for generating attributed text generation data.
- arXiv: Incorporated as an additional data source to demonstrate the framework's extensibility and improve performance.
- ASQA, ELI5, PopQA, FanOutQA: Standard benchmarks used for evaluating the performance of the trained models.
- New Dataset Generation: The core contribution is the automatic generation of attributed Q&A pairs for SFT and PO stages, eliminating the need for human annotation. The paper reports generating 31,823 initial samples from Wikipedia, which are then filtered.
Key Insights and Ideas
- Automating Data Generation is Key: The scarcity of human-annotated data is a major bottleneck. 𝒜³ demonstrates that high-quality attributed text generation data can be automatically generated, making the task more accessible.
- Quality Over Quantity: The study highlights that a high filtering threshold (ensuring data quality) is more impactful than simply increasing the number of generated samples.
- Robustness through Noise: Introducing "noisy" documents during data generation makes the LLMs more resilient to irrelevant information in real-world retrieval scenarios.
- Preference Optimization is Crucial: The PO stage, which trains on preferred vs. dispreferred responses, significantly boosts citation quality and relevance.
- Generalizability: The framework shows consistent improvements across various backbone LLMs (LLaMA, Mistral), indicating its broad applicability.
Key Results
- Mistral-7B with 𝒜³ achieved a citation recall of 84.4% and precision of 87.0% on the ASQA dataset, significantly outperforming GPT-4 (73.0% recall, 76.5% precision).
- The framework improved citation performance for all backbone models tested across ASQA, ELI5, and PopQA datasets.
- Data filtering and noisy document augmentation strategies were shown to effectively reduce the percentage of irrelevant citations.
- Human evaluation of 100 samples showed 84% were relevant and 78% were supportive, confirming the high quality of the generated data.
Limitations and Future Work
- Hallucination and Bias: Despite efforts, the generated data and trained models may still exhibit hallucination and bias inherited from the backbone LLMs or the corpus. This needs careful addressing before real-world application.
- Language Morphology: Experiments were primarily conducted on English benchmarks. The effectiveness on languages with varied morphology needs further confirmation.
- Backbone Model Diversity: While tested on several LLMs, the framework's effectiveness on a wider range of LLMs, including those with different architectures or sizes, requires further investigation.
- Data Source Expansion: The authors suggest expanding the use of closed-source or open-source LLMs beyond GPT-3.5/GPT-4 for data generation.
- Integration with RLVR: The paper proposes integrating 𝒜³ with Reinforcement Learning with Verifiable Rewards (RLVR) methods as a promising future direction.
Summary updated