Abstract
Knowledge-Enhanced Large Language Models (K-LLMs) enhance Large Language Model (LLM) abilities using external knowledge. Existing K-LLMs toolkits mainly focus on free-textual knowledge, lacking support for heterogeneous knowledge like tables and knowledge graphs, and fall short in comprehensive datasets, models, and user-friendly experience. To address this gap, we introduce KMatrix: a flexible heterogeneous knowledge enhancement toolkit for LLMs including verbalizing-retrieval and parsing-query methods. Our modularity and control-logic flow diagram design flexibly supports the entire lifecycle of various complex K-LLMs systems, including training, evaluation, and deployment. To assist K-LLMs system research, a series of related knowledge, datasets, and models are integrated into our toolkit, along with performance analyses of K-LLMs systems enhanced by different types of knowledge. Using our toolkit, developers can rapidly build, evaluate, and deploy their own K-LLMs systems.
Introduction
While Large Language Models (LLMs) are powerful, they often suffer from "hallucinations" and struggle with knowledge-intensive tasks. Retrieval-Augmented Generation (RAG) helps by providing external context, but existing toolkits face three major hurdles:
- Text-Centricity: Most tools only handle free-text, ignoring the wealth of information stored in structured formats like tables and Knowledge Graphs (KGs).
- Rigidity: They often lack support for "adaptive" methods—systems that can autonomously decide when to look for information and where to find it.
- Lifecycle Gaps: Many tools focus only on deployment, leaving researchers to find separate solutions for training and evaluation.
KMatrix aims to fill these gaps by providing a unified, customizable environment for the entire K-LLM lifecycle, as compared to existing solutions in the table below.
System Architecture and Methods
KMatrix uses a modular architecture where users can connect 16 different types of components (such as Retrievers, Query Parsers, and Generators) using a control-logic flow diagram. This allows for complex system designs involving loops, branches, and conditional logic.
The toolkit supports two primary pathways for handling diverse knowledge:
- Verbalizing-Retrieval: This method converts structured data (tables, KGs) into unified text fragments, allowing standard text-based retrievers to search across all formats simultaneously.
- Parsing-Query: This approach uses a "Query Parser" to generate specific query statements (like SPARQL for KGs or natural language for web APIs) to interact directly with structured databases.
Crucially, KMatrix integrates Adaptive Enhancement. It includes mechanisms for Retrieval Timing Judgment (deciding if retrieval is actually necessary for a given prompt) and Knowledge Source Selection (dynamically choosing the best database to query based on the question's domain).
Integrated Resources and Datasets
KMatrix integrates a massive library of resources to lower the barrier to entry for researchers:
- 11 Knowledge Bases: Including Wikipedia (text), Wikidata (KG), and Wikitable (structured tables).
- 22 Datasets: Categorized into three classes:
RETRIEVE_EVAL(8 datasets for testing retrieval accuracy),ODQA_EVAL(6 datasets for end-to-end performance), andODQA_EVAL_Simplified(4 domain-specific datasets for testing source selection).
Key Insights and Results
Experimental results using KMatrix reveal several critical insights into the behavior of K-LLMs:
- Heterogeneity and Noise: While adding tables and KGs generally improves performance, it also introduces noise. Joint enhancement with all three types sometimes performed slightly worse than text+KG alone on specific datasets due to "noisy" table data.
- The Power of Adaptivity: Adaptive methods like Self-RAG (which uses special tokens to control retrieval) significantly outperformed "Naive-RAG" and iterative "Interleave" methods. This suggests that knowing when to stop searching is as important as knowing where to look.
- LLMs as Librarians: The research demonstrates that LLMs can effectively act as "selectors," autonomously choosing the most relevant domain-specific knowledge interface (e.g., choosing a medical database for a health query) to improve final answer accuracy.
In end-to-end QA tests, the Self-RAG implementation within KMatrix achieved the best results. For example, on the PopQA dataset, Self-RAG reached an accuracy of 61.37%, compared to just 27.51% for a standard Naive-RAG setup using only Wikipedia text.
Conclusion and Future Work
KMatrix provides a comprehensive, open-source toolkit designed to streamline the building and evaluation of complex K-LLMs. The authors identify three main areas for future improvement:
- Efficiency: Optimizing retriever speed to reduce latency when searching through excessively large knowledge bases.
- Data Quality: Implementing automated "knowledge denoising" components to handle noise in sources like Wikitable.
- Continuous Integration: Continually integrating new adaptive methods as they emerge in the research community.
The toolkit and resources are available on GitHub.
Summary updated