Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators

Liang Chen, Yang Deng, Yatao Bian, Zeyu Qin, Bingzhe Wu, Tat-Seng Chua, Kam-Fai Wong

TL;DR

This paper introduces CONNER, a comprehensive framework for evaluating knowledge generated by Large Language Models (LLMs). Surprisingly, they find that while LLM-generated knowledge is less factual than traditional retrieval methods, its relevance and coherence are more crucial for downstream task performance, and factuality errors have a smaller impact than expected.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly used to generate knowledge for tasks like question answering and dialogue. While they often outperform traditional information retrieval systems, there's a significant concern about the factuality and potential implications of this generated knowledge. Existing evaluation methods are often subjective, time-consuming, or rely on human-annotated references, limiting their scalability and applicability. This paper aims to address this gap by proposing a systematic, automatic, and reference-free framework to evaluate LLM-generated knowledge across multiple dimensions.

Methods

The core contribution is the CONNER framework, which evaluates generated knowledge from six perspectives: Factuality, Relevance, Coherence, Informativeness, Helpfulness, and Validity. These are categorized into:

  • Intrinsic Evaluation: Assesses the internal properties of the knowledge itself.
  • Factuality: Verified against external evidence (e.g., search engine results, Wikipedia).
  • Relevance: Measured by how well the knowledge relates to the user's query.
  • Coherence: Evaluated at both sentence and paragraph levels, assessing fluency and logical flow.
  • Informativeness: Determines how novel or unexpected the knowledge is compared to the model's existing knowledge.
  • Extrinsic Evaluation: Assesses the impact of the knowledge on downstream tasks.
  • Helpfulness: Measures how much the generated knowledge aids in answering a query or completing a task.
  • Validity: Evaluates the factuality of the final answer produced using the generated knowledge.

CONNER is designed to be reference-free, meaning it doesn't require pre-existing ground truth knowledge for evaluation.

Datasets

The authors evaluated their framework on two widely-used knowledge-intensive benchmark datasets:

  • Natural Questions (NQ): An open-domain question answering dataset.
  • Wizard of Wikipedia (WoW): A knowledge-grounded dialogue dataset.

They randomly sampled 500 examples from the test sets of each dataset for their experiments.

Key Insights and Ideas

  • Factuality isn't everything: Surprisingly, lower factuality in LLM-generated knowledge didn't significantly harm downstream task performance. This suggests that small factual inaccuracies are less detrimental than expected.
  • Relevance and Coherence are King: The study highlights that relevance and coherence of the generated knowledge are more critical for successful downstream task completion than perfect factuality.
  • LLMs Outperform Retrieval (Mostly): LLMs generally surpassed traditional retrieval methods (like DPR) in most evaluation aspects, except for factuality and informativeness.
  • Reference-Free Evaluation is Viable: CONNER's success in correlating with human judgments without relying on reference knowledge is a significant step towards scalable and practical evaluation.
  • Model Size Doesn't Guarantee Quality: Larger LLMs don't always outperform smaller ones in intrinsic evaluation metrics, but they tend to perform better in extrinsic (downstream) evaluations.
  • Long-tail and Long-form Challenges: LLMs exhibit lower reliability when generating rare or "long-tail" knowledge and tend to be more error-prone when generating longer pieces of text. Concise generation is preferred for factuality.

Key Results

  • LLMs (ChatGPT, LLaMA) generally achieved higher scores than DPR (a retrieval model) on Relevance, Coherence, Helpfulness, and Validity across both NQ and WoW datasets.
  • While LLMs showed lower factuality scores compared to DPR, their generated knowledge still led to higher validity in downstream tasks.
  • CONNER metrics demonstrated a strong correlation with human evaluations, outperforming baseline reference-reliant metrics.
  • The paper proposes two strategies using CONNER to improve LLM performance: Prompt Engineering (selecting better few-shot examples) and Knowledge Selection (choosing the best generated knowledge from multiple options). These strategies showed relative improvements in helpfulness and intrinsic quality.

Limitations and Future Work

  • Wikipedia Dependency: The evaluation primarily relies on Wikipedia as the source of truth. Future work could expand to a broader range of world knowledge.
  • Language Generalization: The framework's NLP components might pose challenges for non-English languages.
  • Sentence-Level Factuality: The current factuality assessment is at the sentence level. Future research could focus on more fine-grained detection of factual errors, especially concerning numbers, times, and fabricated concepts.
  • Broader Knowledge Sources: The authors suggest expanding evaluations to include more diverse knowledge sources beyond Wikipedia.
  • Fine-grained Error Detection: Developing more sophisticated methods for detecting specific types of factual inaccuracies is a promising direction.

Summary updated

Abstract

Large language models (LLMs) outperform information retrieval techniques for downstream knowledge-intensive tasks when being prompted to generate world knowledge. However, community concerns abound regarding the factuality and potential implications of using this uncensored knowledge. In light of this, we introduce CONNER, a COmpreheNsive kNowledge Evaluation fRamework, designed to systematically and automatically evaluate generated knowledge from six important perspectives – Factuality, Relevance, Coherence, Informativeness, Helpfulness and Validity. We conduct an extensive empirical analysis of the generated knowledge from three different types of LLMs on two widely studied knowledge-intensive tasks, i.e., open-domain question answering and knowledge-grounded dialogue. Surprisingly, our study reveals that the factuality of generated knowledge, even if lower, does not significantly hinder downstream tasks. Instead, the relevance and coherence of the outputs are more important than small factual mistakes. Further, we show how to use CONNER to improve knowledge-intensive tasks by designing two strategies: Prompt Engineering and Knowledge Selection. Our evaluation code and LLM-generated knowledge with human annotations will be released to facilitate future research.
BibTeX
@inproceedings{chen2023beyond,
  author = {Liang Chen and Yang Deng and Yatao Bian and Zeyu Qin and Bingzhe Wu and Tat-Seng Chua and Kam-Fai Wong},
  title = {{Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.390},
}
Figure