Application of Generative AI as an Enterprise Wikibase Knowledge Graph Q&A System

Renê Mendes, Dimas Oliveira, Victor GarciaACL 2024

TL;DR

This paper presents an enterprise-grade Knowledge Graph (KG) Q&A system built using Generative AI and Wikibase. It addresses the challenge of making complex organizational data accessible through natural language, enabling users to query information without needing technical expertise in graph databases or query languages.

What Problem Does This Paper Address?

Organizations are awash in data, but extracting meaningful insights often requires specialized skills. Enterprise Knowledge Graphs (EKGs) offer a structured way to represent this data, but querying them typically involves complex languages like SPARQL. Generative AI, particularly Large Language Models (LLMs), has shown promise in understanding natural language, but integrating them with private enterprise data securely and effectively is a significant challenge. This paper tackles how to bridge this gap by creating a system that allows users to ask questions in plain English about an organization's data, stored in an EKG, and receive accurate, context-aware answers.

Methods

The core of the system is a Retrieval-Augmented Generation (RAG) approach. This involves:

  • Enterprise Knowledge Graph (EKG) Construction: The authors utilize Wikibase, an open-source software known for its flexibility and out-of-the-box services, to build the EKG. This includes defining ontologies (formal representations of concepts and their relationships) based on the Basic Formal Ontology (BFO) and proprietary ontologies tailored to the organization's domain (e.g., academic, health, security).
  • Synthetic Data Generation: To protect sensitive information, synthetic data was generated using GPT-3, mimicking real-world academic and customer data. This synthetic data was then loaded into the EKG.
  • Generative AI Q&A System: A Langchain-based Re-Act (Reasoning-Action) agent is employed. This agent can reason about user queries, use tools to interact with the EKG (specifically, Python functions to query Wikidata and the private EKG via SPARQL endpoints), and then translate the results into natural language answers. The system was tested with Gemini Pro 1.0 and can be extended to other LLMs.

Datasets

  • Synthetic Data: The primary dataset used for development and testing was synthetically generated student and patient data. This was created using GPT-3 to mimic data from an academic data warehouse and a Customer Relationship Management (CRM) system, ensuring privacy.
  • Wikidata: The system was also connected to Wikidata, a large, publicly available knowledge graph, for comparative analysis and to leverage its extensive data for certain query types.

Key Insights and Ideas

  • Wikibase as a Practical Choice: Wikibase proved advantageous for rapid prototyping due to its readily available services like SPARQL endpoints and graphical interfaces, enabling a functional sandbox environment in just a few days.
  • RAG for Enterprise Data: The RAG approach, combining LLMs with a private EKG, is crucial for grounding AI responses in specific organizational data, mitigating LLM "hallucinations" and ensuring relevance.
  • Synthetic Data for Privacy: Using synthetically generated data is a viable strategy for developing and testing KG-based AI systems in sensitive enterprise environments without compromising data privacy.
  • Domain-Specific EKGs Outperform General KGs for Specific Tasks: The Re-Act agent performed significantly better when querying the private EKG compared to Wikidata for descriptive questions, highlighting the benefit of a focused, domain-specific knowledge base.

Key Results

The paper reports on the performance of the KG Q&A system across three types of questions:

  • Type 1 (Descriptive): The system achieved a higher success rate when connected to the private EKG (80%) compared to Wikidata (40%).
  • Type 2 (Property Linking): The success rate was 71% with the private EKG and 50% with Wikidata.
  • Type 3 (Calculations): The success rate was 38% with the private EKG and not explicitly stated for Wikidata, but implied to be lower.

The authors note that the discrepancy in Type 2 questions might be due to Gemini Pro's prior exposure to Wikidata data.

Limitations and Future Work

  • Privacy and Access Control: While Wikibase is open-source, managing user-specific data access within a corporate environment requires custom extensions or development, as Wikidata itself doesn't inherently support granular access control.
  • LLM Robustness: The system's performance, particularly in generating SPARQL queries, can be improved by using more robust LLMs (like Gemini 1.5 Pro or GPT-4) and fine-tuning them.
  • Production Deployment: For production, premium LLM versions will be used to guarantee that private data is not used for model training.
  • Future Work: The authors plan to implement a production-ready approach, focusing on advanced LLMs, fine-tuning for SPARQL generation, and robust access control mechanisms for the EKG. They also suggest exploring Wikibase's out-of-the-box data visualization and horizontal scaling capabilities.

Summary updated

BibTeX
@inproceedings{mendes2024application,
  author = {Renê Mendes and Dimas Oliveira and Victor Garcia},
  title = {{Application of Generative AI as an Enterprise Wikibase Knowledge Graph Q\&A System}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure