ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA Datasets with Large Language Models

Ronak Pradeep, Daniel Lee, Ali Mousavi, Jeffrey Pound, Yisi Sang, Jimmy Lin, Ihab F. Ilyas, Saloni Potdar, Mostafa Arefiyan, Yunyao Li

TL;DR

The paper introduces ConvKGYarn, a novel method for automatically generating large-scale, configurable conversational Knowledge Graph Question Answering (KGQA) datasets. This addresses the challenge of creating dynamic datasets that keep pace with evolving user needs and LLM capabilities, enabling more robust evaluation of conversational AI.

What Problem Does This Paper Address?

The rapid rise of Large Language Models (LLMs) and conversational assistants has created a critical need for high-quality, dynamic datasets to train and evaluate these systems. Existing conversational KGQA datasets, often human-curated, struggle to scale and adapt to the ever-changing information landscape and diverse user interaction styles (text vs. voice, with or without disfluencies, typos, etc.). This makes it difficult to rigorously assess LLMs' ability to handle real-world conversational queries grounded in structured knowledge.

Methods

ConvKGYarn employs a multi-stage pipeline to generate conversational KGQA datasets:

  1. KG Predicate Extraction: It starts by extracting relevant predicates (properties) for specific entity types from a knowledge graph (Wikidata).
  2. LLM Predicate Selection: An LLM (GPT-4) is used to filter these predicates, selecting only those that are most "interesting" and suitable for conversational QA, aiming for richness and contextual appropriateness.
  3. Related Entity Generation: The system identifies and incorporates related entities to enrich the dataset with diverse, relevant information that might not be directly linked to the primary entity.
  4. Fact Extraction: Factual information is extracted from the KG for the selected entities and predicates, including handling qualified facts (facts with additional context like timestamps).
  5. Synthetic Question Template Generation: This is a core innovation. ConvKGYarn uses LLMs to generate question templates that mimic various human interaction styles:
  6. Text/Search: Short, keyword-based queries.
  7. Voice: More natural, potentially with deixis (referencing previous turns) and disfluencies (speech errors).
  8. Typo Augmentation: Introduces realistic spelling errors.
  9. Qualified Facts: Generates questions that account for qualifiers. The system generates multiple variants for each fact and linguistic phenomenon to ensure comprehensive coverage.

  10. Conv. Factoid QA Instance Creation: The generated templates are slot-filled with extracted facts and related entities to create conversational instances.

Datasets

The paper introduces ConvKGYarn as a new method for generating datasets. They present two main datasets derived from this method:

  • General Set: This is a large-scale dataset comprising 29 million entities and 196 million facts, derived from a filtered Wikidata dump. It generates 24 question variants per fact across different interaction styles.
  • Related Set: A more specialized dataset focusing on "Human-type" entities, with 210,000 entities and 6.1 million facts. It offers a higher density of question variants per fact (average of 54) by incorporating related entity-specific follow-up queries.

These datasets are designed to be significantly larger and more configurable than existing human-curated datasets like ConvQuestions.

Key Insights and Ideas

  • LLMs as Dataset Generators: The paper demonstrates that LLMs can be effectively leveraged not just for answering questions but also for generating high-quality, diverse, and configurable datasets, reducing reliance on expensive human annotation.
  • Configurability is Key: The ability to generate datasets with specific interaction styles (voice, text, typos, deixis, disfluencies) is crucial for evaluating LLMs in a controlled, "confounder-free" manner. This allows researchers to isolate the impact of specific linguistic phenomena.
  • Human-like Data is Achievable: Through careful prompting and pipeline design, ConvKGYarn can produce datasets that are psychometrically comparable to human-curated ones, challenging the notion that synthetic data is inherently lower quality.
  • LLMs Struggle with Fact Recall: The evaluation of LLMs on these datasets highlights their limitations in parametric knowledge recall, underscoring the need for retrieval-augmented systems.
  • Deixis and Related Entities Improve Quality: Human evaluations suggest that incorporating deixis and related entities leads to more natural and relevant conversations.

Key Results

  • Dataset Scale: The General set contains 29 million entities and 196 million facts, with 24 question variants per fact, far exceeding the scale of human-curated datasets.
  • Human Evaluation: Psychometric evaluations showed that ConvKGYarn-generated data achieved high scores for Fluency (75.5%), Relevance (62.2%), Diversity (56.0%), and Grammar (56.6%) in the Single-Model Rating, and nearly matched human-curated datasets in pairwise comparisons.
  • LLM Performance: LLM evaluations (GPT-3.5 and GPT-4) using the ConvKGYarn datasets revealed that GPT-4 generally outperformed GPT-3.5, and that typos significantly degraded LLM performance, while disfluencies had a negligible impact.
  • Annotator Agreement: High inter-annotator agreement (70.53% for psychometric evaluation, 85.6% for comparative analysis) validates the quality and reliability of the human evaluations.

Limitations and Future Work

  • LLM Sensitivity to Typos: The paper notes that LLMs are sensitive to typos, which negatively impacts their performance. This suggests a need for more robust typo handling in future LLM development.
  • Deixis and Disfluency Impact: While the impact of disfluencies was surprisingly minimal, the study suggests further investigation into how LLMs handle various linguistic phenomena in conversational settings.
  • Scalability of Human Evaluation: While ConvKGYarn excels at generating data at scale, human evaluation remains a bottleneck. Future work could explore more efficient or automated evaluation methods.
  • Refinement of Predicate Selection: While LLMs are used for predicate selection, further research could explore more sophisticated methods to ensure optimal predicate choices for conversational contexts.
  • Broader KG Integration: The framework could be extended to incorporate other types of knowledge representations beyond standard KGs.

Summary updated

Abstract

The rapid evolution of Large Language Models (LLMs) and conversational assistants necessitates dynamic, scalable, and configurable conversational datasets for training and evaluation.These datasets must accommodate diverse user interaction modes, including text and voice, each presenting unique modeling challenges. Knowledge Graphs (KGs), with their structured and evolving nature, offer an ideal foundation for current and precise knowledge.Although human-curated KG-based conversational datasets exist, they struggle to keep pace with the rapidly changing user information needs.We present ConvKGYarn, a scalable method for generating up-to-date and configurable conversational KGQA datasets. Qualitative psychometric analyses demonstrate ConvKGYarn’s effectiveness in producing high-quality data comparable to popular conversational KGQA datasets across various metrics.ConvKGYarn excels in adhering to human interaction configurations and operating at a significantly larger scale.We showcase ConvKGYarn’s utility by testing LLMs on diverse conversations — exploring model behavior on conversational KGQA sets with different configurations grounded in the same KG fact set.Our results highlight the ability of ConvKGYarn to improve KGQA foundations and evaluate parametric knowledge of LLMs, thus offering a robust solution to the constantly evolving landscape of conversational assistants.
BibTeX
@inproceedings{pradeep2024convkgyarn,
  author = {Ronak Pradeep and Daniel Lee and Ali Mousavi and Jeffrey Pound and Yisi Sang and Jimmy Lin and Ihab F. Ilyas and Saloni Potdar and Mostafa Arefiyan and Yunyao Li},
  title = {{ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA Datasets with Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-industry.89},
}
Figure