An LLM-Based QA System for Chinese Painting and Calligraphy with Knowledge Graphs and External Documents

Jing Wan, Xinrong Li, Hao Zhang, Ao Zou, R. WangResearch Square 2025 · DOI

Abstract

Chinese Painting and Calligraphy (ChP&C), key elements of Chinese cultural heritage, hold rich historical and artistic value. Although Large Language Models (LLMs) excel in open-domain question answering (QA), they often suffer from hallucinations in domain-specific contexts. To address this, we propose a ChP&C QA method integrating LLMs with a retrieval-augmented approach using a Knowledge Graph (KG) and external documents. The method decomposes complex questions into sub-questions and entities, retrieves relevant knowledge from KG and documents, and integrates answers via a dedicated module. We constructed a QA dataset focused on ChP&C to validate the proposed method. Additionally, a QA system was developed to systematically evaluate its performance in real-world applications. Experimental results demonstrate improved semantic understanding and answer accuracy by effectively combining structured and unstructured information. This system offers a reliable tool for accessing ChP&C knowledge and serves as a reference for intelligent QA in other cultural heritage domains.

Introduction

While Large Language Models like GPT-4 are impressive generalists, they often struggle with "hallucinations"—generating confident but false information—when asked about highly specialized domains. Chinese Painting and Calligraphy (ChP&C) is a prime example of such a domain. Its knowledge is deeply fragmented across historical texts, academic research, and museum records.

Current search engines rely heavily on keyword matching, which fails to capture deep semantic relationships (e.g., "Which artist from the Song Dynasty influenced this specific 17th-century style?"). Furthermore, existing LLMs often lack the specific, up-to-date, or niche data required to provide professional-grade answers in this field. This research bridges the gap by creating a system that "grounds" the LLM in a verified, domain-specific knowledge base.

Methodology

The proposed hybrid Retrieval-Augmented Generation (RAG) framework follows a four-step pipeline designed to maximize accuracy and context:

  1. Question Decomposition: Using GPT-3.5-turbo to break a complex, multi-hop question into simpler sub-questions and identify key entities.
  2. Knowledge Graph (KG) Retrieval: The system queries a specialized ChP&C Knowledge Graph to extract structured facts. It supports up to three-hop expansions to find non-obvious connections.
  3. External Document Retrieval: Using the BM25 algorithm, the system searches a corpus of nearly 9,000 cleaned documents (crawled from sources like the Palace Museum and Baidu Baike) to find relevant paragraphs that provide context the KG might lack.
  4. Answer Integration: A final LLM module takes the candidate answers and evidence from both the KG and the documents, performs a "comprehensive judgment," and synthesizes a final, coherent response.
The overall architecture of the ChP&C QA system, illustrating the integration of LLM reasoning with KG and document retrieval.

Dataset Construction: CHPCQAD

A critical contribution of this work is the CHPCQAD (Chinese Painting and Calligraphy QA Dataset). To create a high-quality benchmark, the researchers used a two-stage method. First, "Asking" and "Locating" templates generated logically sound questions represented in KoPL (a programmatic logical form). Then, an LLM was used to rewrite these into natural, fluent human language. This hybrid approach ensures both logical rigor and linguistic variety.

Distribution of query types within the CHPCQAD dataset, showing the variety of reasoning tasks covered.

The dataset contains 4,055 samples and covers nine query types, including multi-hop reasoning, comparisons, and factual verification. It is supported by an external corpus of 8,949 structured documents related to ChP&C entities.

Key Results and Insights

The proposed system was tested against several competitive baselines, including RNN, RGCN, and KVMemNet. The results highlight the effectiveness of the multi-source retrieval approach:

  • Superior Accuracy: The system achieved an overall accuracy of 50.65%, more than doubling the performance of many traditional deep learning models which typically scored between 12% and 35%.
  • Strong Generalization: In zero-shot scenarios with unseen questions, the model maintained a 52.90% accuracy.
  • Complementary Data: Ablation studies revealed that the Knowledge Graph is essential for precise facts and multi-hop reasoning, while external documents are vital for handling descriptive, nuanced queries.

Conclusion and Future Work

By combining structured Knowledge Graphs with unstructured external documents, this system significantly reduces LLM hallucinations in the specialized field of Chinese art history. Future research will focus on aligning knowledge from diverse decentralized sources and expanding the system into a multimodal framework. Incorporating image data will allow the system to "see" the paintings and calligraphy it discusses, providing a truly comprehensive tool for cultural heritage research.

Summary updated

Abstract

<title>Abstract</title> Chinese Painting and Calligraphy (ChP&amp;C), key elements of Chinese cultural heritage, hold rich historical and artistic value. Although Large Language Models (LLMs) excel in open-domain question answering (QA), they often suffer from hallucinations in domain-specific contexts. To address this, we propose a ChP&amp;C QA method integrating LLMs with a retrieval-augmented approach using a Knowledge Graph (KG) and external documents. The method decomposes complex questions into sub-questions and entities, retrieves relevant knowledge from KG and documents, and integrates answers via a dedicated module. We constructed a QA dataset focused on ChP&amp;C to validate the proposed method. Additionally, a QA system was developed to systematically evaluate its performance in real-world applications. Experimental results demonstrate improved semantic understanding and answer accuracy by effectively combining structured and unstructured information. This system offers a reliable tool for accessing ChP&amp;C knowledge and serves as a reference for intelligent QA in other cultural heritage domains.
BibTeX
@inproceedings{wan2025llm,
  author = {Jing Wan and Xinrong Li and Hao Zhang and Ao Zou and R. Wang},
  title = {{An LLM-Based QA System for Chinese Painting and Calligraphy with Knowledge Graphs and External Documents}},
  booktitle = {Research Square},
  year = {2025},
  doi = {10.21203/rs.3.rs-6824315/v1},
}
Figure