Evaluating Design Choices in Verifiable Generation with Open-source Models

Shuyang Cao, Lu WangNAACL 2025

TL;DR

This paper investigates how to make open-source Large Language Models (LLMs) better at "verifiable generation" – producing text with accurate citations to supporting documents. They found that a direct generation pipeline, fine-tuned with Direct Preference Optimization (DPO) and diverse training data, works best. While reranking can help, it's not always cost-effective for top-performing models.

What Problem Does This Paper Address?

Verifiable generation is crucial for building user trust in LLMs by allowing them to check generated claims against cited sources. While proprietary models excel at this, open-source LLMs often struggle with accurate citations, especially when using in-context learning. This paper aims to identify the key design choices that can improve verifiable generation capabilities in open-source LLMs, focusing on generation pipelines, fine-tuning strategies, and inference-time techniques.

Methods

The authors explored three generation pipelines:

  • Direct Generation: The LLM generates the final output with citations in a single step.
  • Decomposed Generation: The task is split into two stages: a content generation model and a separate citation generation model.
  • Joint Generation: A hybrid approach where an LLM generates both intermediate content and final citations in a single pass.

They investigated several fine-tuning strategies:

  • Supervised Fine-Tuning (SFT): Standard fine-tuning on existing data.
  • Rejection Sampling (RJS): Fine-tuning on the best-performing outputs from a set of generated samples.
  • Direct Preference Optimization (DPO): A method that directly optimizes the LLM based on pairwise comparisons of preferred and rejected outputs, without needing a separate reward model.

They also examined the impact of preference data construction, varying the diversity of content and citations in training samples. Finally, they evaluated inference-time compute techniques, specifically over-generation followed by reranking.

Datasets

  • SCIFI: A citation-rich dataset with subsentence-level citations sourced from Wikipedia, used for training and preference data collection.
  • ALCE: A question-answering dataset with retrieved documents, used for evaluating generalizability. The ASQA and ELI5 subsets were specifically used.

Key Insights and Ideas

  • Direct is often best: The simplest approach, direct generation, surprisingly outperforms more complex decomposed or joint pipelines for verifiable generation. This might be because it aligns better with how LLMs are pre-trained.
  • DPO shines: Direct Preference Optimization (DPO) consistently yields better results than other fine-tuning methods, especially when compared to supervised fine-tuning or rejection sampling.
  • Diversity matters for DPO: For DPO to be most effective, the preference data needs sufficient content diversity. Simply generating more citations with the same content doesn't improve performance as much.
  • Reranking offers a boost, but at a cost: Inference-time reranking can improve output quality across the board, but the gains are marginal for already well-tuned models, raising questions about its cost-effectiveness.
  • Open-source models are catching up: The strong performance of newer open-source models (like Llama-3.1) on verifiable generation, especially after fine-tuning, indicates progress in this area.

Key Results

  • Pipeline Performance: Direct generation consistently achieved better or comparable combined quality across all tested open-source LLMs (Llama-3.1-8B, Mistral-Nemo, Qwen-2.5-7B, Phi-3.5-Mini).
  • Fine-tuning Strategy: DPO outperformed SFT and RJS across various pipelines and models. For example, on Llama-3.1-8B with direct generation, DPO achieved a combined quality score of 18.56, compared to 16.31 for RJS and 8.05 for ICL.
  • Content Diversity: For DPO fine-tuning, configurations that balanced content and citation diversity performed better. For instance, using outputs based on two distinct intermediate content outputs was better than using outputs with many different citations but the same content.
  • Inference-Time Compute: Reranking consistently improved quality for all systems, but the improvement was smaller for DPO-tuned models.
  • Generalizability: Fine-tuning with DPO on SCIFI showed good generalizability to the out-of-domain ALCE dataset, particularly for citation quality. However, out-of-domain fine-tuning could slightly decrease content quality on in-domain data.

Limitations and Future Work

  • Candidate Source Handling: The paper acknowledges that handling the pool of candidate cited sources could be more sophisticated, potentially involving multi-turn processing or dense representations.
  • Dynamic Source Retrieval: The current experiments use fixed candidate sources. Real-world scenarios involve dynamic retrieval from search engines, which introduces more variability and could impact model performance unpredictably.
  • Source Reliability: The study used reliable sources like Wikipedia. In practical settings, LLMs might encounter less credible or biased sources, posing a risk of propagating misinformation. Future work should investigate robustness in uncontrolled environments and explore content filters.
  • Cost-Effectiveness of Reranking: While reranking improves performance, its marginal gains for top models might not justify the computational cost. Future research could explore more efficient reranking strategies.

Summary updated

BibTeX
@inproceedings{cao2025evaluating,
  author = {Shuyang Cao and Lu Wang},
  title = {{Evaluating Design Choices in Verifiable Generation with Open-source Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure