Knowledge Acquisition through Continued Pretraining is Difficult: A Case Study on r/AskHistorians

Jan Hoffbauer, Sylwester Sawicki, Marc Ulrich, Tolga Buz, Konstantin Dobler, Moritz Schneider, Gerard De MeloACL 2024

TL;DR

This paper investigates the effectiveness of common techniques like supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) for injecting domain-specific knowledge into Large Language Models (LLMs). Surprisingly, they found that these methods, when applied to a rich historical dataset from r/AskHistorians, did not improve knowledge accuracy and, in some cases, even decreased it, while negatively impacting stylistic quality.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly used by knowledge workers, but they often struggle with niche, domain-specific knowledge, leading to factual inaccuracies or "hallucinations." While techniques like instruction tuning, alignment, and retrieval-augmented generation exist to improve LLM knowledge, they each have drawbacks. This paper specifically addresses the challenge of how to effectively ingest and evaluate domain-specific knowledge into LLMs, particularly for professional applications where accuracy is paramount. They focus on the difficulty of this process and explore whether continued pretraining or fine-tuning on a dense, real-world dataset can overcome LLMs' inherent limitations in specialized domains.

Methods

The researchers employed a multi-faceted approach to investigate knowledge acquisition in LLMs:

  • Dataset Curation: They curated a high-quality dataset from the r/AskHistorians subreddit, known for its in-depth historical discussions and strict moderation. This dataset was filtered to ensure relevance and quality.
  • Model Training: They utilized leading open-source LLMs (Zephyr-7B-beta, Phi-1.5, Pythia-1.4B, Mistral-7B-v0.1) and applied two primary training techniques:
  • Supervised Fine-Tuning (SFT): Used to teach the model specific tasks and response styles.
  • Direct Preference Optimization (DPO): A more efficient alignment technique that uses preference scores directly.
  • They also experimented with LoRA (Low-Rank Adaptation) for resource-efficient fine-tuning and full-weight fine-tuning to compare their impact.
  • Knowledge Filling Task: To specifically evaluate knowledge accuracy without linguistic bias, they created a "Knowledge Filling" dataset. This involves rephrasing facts from the dataset into cloze-style questions, where the LLM is prompted to fill in a single missing fact.
  • Evaluation Metrics: Beyond the Knowledge Filling task, they also measured:
  • NLP Metrics: Perplexity, text complexity, reading time, and toxicity.
  • Human and LLM-based Preference Evaluation: Comparing model outputs using both human annotators and GPT-4 as a judge to assess overall quality and preference.

Datasets

  • r/AskHistorians Dataset: The primary dataset used for training and evaluation, curated from the r/AskHistorians subreddit. It contains questions and long-form answers on historical topics.
  • Knowledge Filling Dataset: A newly introduced benchmark created by the authors for evaluating knowledge accuracy. It consists of cloze-style questions derived from the r/AskHistorians dataset.
  • UltraChat and UltraFeedback: Used as supplementary datasets for continual learning during SFT and DPO to prevent the model from forgetting its original generation qualities.

Key Insights and Ideas

  • Continued Pretraining vs. Fine-tuning: The study suggests that initial pretraining has a much stronger effect on knowledge accuracy than continued pretraining or fine-tuning, especially when the fine-tuning data is not perfectly aligned or when overfitting occurs.
  • The Paradox of Fine-Tuning: Counter-intuitively, fine-tuning on the r/AskHistorians dataset decreased knowledge accuracy for some models, even when the evaluation questions were derived from the training data. This implies that simply exposing an LLM to domain-specific facts during fine-tuning doesn't guarantee improved knowledge recall.
  • LoRA's Potential Impact: While LoRA is a practical technique for fine-tuning large models, the paper hypothesizes that its low-rank nature might hinder knowledge ingestion, though their experiments with full-weight fine-tuning on Phi-1.5 did not yield significantly better results.
  • Stylistic Degradation: Fine-tuning not only failed to improve knowledge accuracy but also negatively impacted stylistic quality. Models became more subjective, less concise, and exhibited increased toxicity and reading time, while text complexity decreased.
  • Overfitting as an Upper Bound: Deliberately overfitting a model to a subset of the data did yield higher knowledge accuracy, but it was still far from perfect (49%), highlighting the difficulty of achieving robust knowledge injection.

Key Results

  • Knowledge Accuracy:
    • Baseline Mistral-7B-v0.1 and Zephyr-7B-beta achieved accuracies of 32% and 31%, respectively, on the Knowledge Filling task.
  • Fine-tuning Zephyr-7B-beta on r/AskHistorians decreased accuracy to 29% (SFT) and 28% (SFT + DPO).
  • Phi-1.5 (not pretrained on Reddit) had 8% accuracy, while Pythia-1.4B (pretrained on Reddit) had 13%. Full-weight SFT on Phi-1.5 resulted in 9% accuracy.
  • Subset-overfit SFT on Zephyr-7B-beta reached 49% accuracy.
  • Stylistic Adaptation:
    • Fine-tuned models showed improved perplexity on the training data, indicating learning of the dataset's linguistic style.
  • However, fine-tuned models exhibited increased toxicity and reading time, and decreased text complexity compared to original Reddit answers.
  • Preference Evaluation:
    • Both human annotators and GPT-4 judges preferred the baseline Zephyr-7B-beta model over the fine-tuned version.
  • GPT-4 as a judge showed a 63% agreement rate with human annotators.
  • When comparing original Reddit answers to the baseline Zephyr-7B-beta, the original answers were preferred in 54% of cases.

Limitations and Future Work

  • Test Set Contamination: There's a risk that some models (e.g., Mistral-7B-v0.1) might have seen parts of the test dataset during their pretraining due to large, uncurated web crawls. The use of Phi-1.5, which was not pretrained on Reddit, helps mitigate this.
  • Dataset Size and Curation: The r/AskHistorians dataset, while high-quality, is relatively small and time-consuming to curate. The authors acknowledge that it may inadvertently contain factual inaccuracies and ambiguities (e.g., different formats for dates or names).
  • Limited Experiment Combinations: Due to computational resource constraints, not all possible combinations of model variants and training techniques were tested.
  • Future Work:
    • Identifying more powerful methods for injecting knowledge into LLMs.
  • Facilitating the creation of similar knowledge benchmarks at a larger scale.
  • Automating the process of compiling lists of possible answers to reduce manual evaluation effort.

Summary updated

BibTeX
@inproceedings{hoffbauer2024knowledge,
  author = {Jan Hoffbauer and Sylwester Sawicki and Marc Ulrich and Tolga Buz and Konstantin Dobler and Moritz Schneider and Gerard De Melo},
  title = {{Knowledge Acquisition through Continued Pretraining is Difficult: A Case Study on r/AskHistorians}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure