KMatrix-2: A Comprehensive Heterogeneous Knowledge Collaborative Enhancement Toolkit for Large Language Model

Shunchuan Wu, Di Wu, Wangtao Sun, Ziyang Huang, Xiaowei Yuan, Kun Luo, XueYou Zhang, Shizhu He, Jun Zhao, Kang Liu2025 · DOI

Abstract: Shun Wu, Di Wu, Wangtao Sun, Ziyang Huang, Xiaowei Yuan, Kun Luo, XueYou Zhang, Shizhu He, Jun Zhao, Kang Liu. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.

2025.

Introduction

Large Language Models (LLMs) have revolutionized natural language processing, yet they remain tethered to two significant limitations: hallucinations and a lack of access to domain-specific, real-time information. While Retrieval-Augmented Generation (RAG) has emerged as a popular solution, existing toolkits often focus narrowly on declarative knowledge (textual facts) while neglecting procedural knowledge (rules and logic).

KMatrix-2 is an open-source toolkit designed to bridge this gap. It provides a modular, flexible framework for integrating heterogeneous knowledge sources—ranging from unstructured PDFs to formal logic rules—enabling LLMs to perform more reliable and complex reasoning tasks.

The Challenge of Heterogeneous Knowledge

Most current K-LLM (Knowledge-enhanced LLM) frameworks suffer from three primary issues:

  1. Limited Knowledge Scope: They typically handle only "declarative" knowledge, ignoring the "how-to" procedural rules essential for logical deduction.
  2. Rigidity: Customizing retrieval workflows often requires extensive hard-coding, making it difficult for researchers to experiment with different patterns.
  3. Knowledge Conflicts: When external information contradicts a model's internal training data or other retrieved sources, standard systems often fail to resolve the discrepancy.

KMatrix-2 addresses these by treating diverse data formats as first-class citizens within a unified pipeline.

The KMatrix-2 Framework

The toolkit breaks the knowledge enhancement process into six distinct modular stages: Knowledge Management, Integration, Retrieval, Pattern Selection, Conflict Resolution, and Generation. This modularity allows users to swap components—such as different retrievers or LLM backends—without rebuilding the entire system.

The modular framework of KMatrix-2, illustrating the pipeline from knowledge management to final task output.

Comprehensive Heterogeneous Support

KMatrix-2 supports a vast array of formats, ensuring that both factual and inferential data can be utilized effectively. This includes traditional documents (Word, PDF, TXT), structured data (Excel, SQLite, JSON), and graph-based knowledge (Neo4j, GraphDB). Crucially, it also incorporates procedural knowledge through natural language rules, first-order logic (FOL), and formal code.

A breakdown of the declarative and procedural knowledge types and formats supported by KMatrix-2.

Technical Innovations

Self-Induction Retrieval

Retrieving procedural knowledge like code or logic rules is difficult because of the "semantic gap" between a natural language query and formal syntax. KMatrix-2 introduces Self-Induction Retrieval, where the LLM "induces" potential rules or logic before the search begins. This pre-processing step helps the retriever identify relevant procedural snippets even when the initial query is vague.

Knowledge Conflict Resolution

KMatrix-2 features dedicated modules to handle contradictions. Inter-Context (IC) resolution manages disagreements between different retrieved snippets, while Context-Memory (CM) resolution helps the model prioritize external, up-to-date facts over its own internal (and potentially outdated) weights.

Flexible Enhancement Patterns

Users can select from four distinct interaction patterns based on the complexity of the task:

  • Naive: Simple one-shot retrieval and generation.
  • Loop: Multi-step decomposition for complex queries.
  • Iterative: Repeated cycles of retrieval and generation to refine answers.
  • Adaptive: The model retrieves information only when it determines its internal knowledge is insufficient.

Key Results and Performance

The toolkit has been validated across diverse benchmarks, including factual QA (TriviaQA, HotpotQA) and procedural retrieval (RuleBench). Key findings include:

  • Procedural Accuracy: On the Clutrr-Fol rule retrieval task, the Self-Induction method improved recall from approximately 5% to 13%.
  • Conflict Resolution: In tests involving Context-Memory conflicts, the toolkit's resolution methods boosted model accuracy from 10% to nearly 64%.
  • Collaborative Gains: Combining declarative and procedural knowledge consistently outperformed declarative-only systems, with accuracy on inferential tasks like Ulogic rising from 49% to 58%.

Conclusion

KMatrix-2 provides a robust foundation for building the next generation of knowledge-aware AI. By integrating procedural logic with factual retrieval and providing explicit mechanisms for conflict resolution, it enables LLMs to move beyond simple pattern matching toward true collaborative reasoning. Future work will focus on expanding support for autonomous "Agentic RAG" and optimizing the toolkit for large-scale industrial deployment.

Summary updated

BibTeX
@misc{wu2025kmatrix,
  author = {Shunchuan Wu and Di Wu and Wangtao Sun and Ziyang Huang and Xiaowei Yuan and Kun Luo and XueYou Zhang and Shizhu He and Jun Zhao and Kang Liu},
  title = {{KMatrix-2: A Comprehensive Heterogeneous Knowledge Collaborative Enhancement Toolkit for Large Language Model}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-demos.42},
}
Figure