Exploring the Role of Knowledge Graph-Based RAG in Japanese Medical Question Answering with Small-Scale LLMs

Yingjian Chen, Feiyang Li, Xingyu Song, Tianxiao Li, Zixin Xu, Xiujie Chen, Issey Sukeda, Irene LiarXiv 2025 · arXiv

TL;DR

This paper presents the first exploration of a Knowledge Graph-based Retrieval-Augmented Generation (KG-RAG) framework tailored for Japanese medical question answering (QA) using small-scale, open-source LLMs. The authors discover that external knowledge injection provides surprisingly limited benefits, as it often introduces specialized noise or simply replicates knowledge the models already inherently possess.

What Problem Does This Paper Address?

While large language models (LLMs) excel at medical QA, deploying commercial giants like GPT-4 in real-world clinical settings is frequently blocked by strict data privacy regulations. This has driven interest in small-scale, open-source LLMs that can be hosted locally.

However, developing medical QA systems for non-English languages faces a steep hurdle: a severe scarcity of high-quality, localized medical training data. To bypass this resource constraint, researchers often turn to Retrieval-Augmented Generation (RAG) to pull in external knowledge. While text-based RAG is common, leveraging structured Knowledge Graphs (KGs) for Japanese medical QA remains largely underexplored. The authors set out to determine if anchoring small-scale LLMs to a medical knowledge graph can overcome the data scarcity problem in low-resource languages.

Methods

Because comprehensive Japanese medical knowledge graphs are difficult to access or non-existent, the authors designed a cross-lingual pipeline that leverages the massive, English-centric Unified Medical Language System (UMLS).

Fig. 1. The pipeline of our knowledge graph-based RAG mechanism. Given a Japanese medical question, relevant medical knowledge is retrieved from the external knowledge base and combined with the original Japanese question as input to the LLM for answer generation. The English translation and full content are provided in Appendix C.

As illustrated in the pipeline above, the framework operates in four distinct steps:

  1. Entity Extraction: A base LLM extracts key medical entities from the user's Japanese query.
  2. Cross-Lingual Retrieval: The extracted Japanese entities undergo word-level translation into English to query the UMLS database, retrieving relevant knowledge graph triples (subject-predicate-object).
  3. Multi-Angle Ranking: A specialized biomedical re-ranker (MedCPT Cross Encoder) scores and selects the top-$k$ most relevant triples based on their semantic similarity to the original question.
  4. Declarative Conversion & Answering: The selected English triples are translated back and converted into natural Japanese declarative sentences by an LLM. These facts are appended to the original prompt as context for the final answer generation.

Datasets

The framework was evaluated on three distinct long-form medical QA datasets, originally curated in English and translated into Japanese using a sophisticated multi-stage agentic machine translation framework:

  • ExpertQA-Bio: 96 high-quality biological questions verified by domain experts.
  • ExpertQA-Med: 504 expert-verified medical questions.
  • LiveQA: 104 test pairs consisting of consumer health questions submitted to the National Library of Medicine.

The dataset statistics are detailed below:

Statistics of the evaluation datasets.

Key Insights and Ideas

The most striking takeaway from this study is that KG-based RAG has a highly volatile and often negligible impact on small-scale LLMs. The authors' case studies revealed a double-edged sword regarding external knowledge injection:

  • The Redundancy Problem: For straightforward or general medical questions, the retrieved KG triples were accurate, but the baseline LLMs already possessed this information internally. Consequently, RAG provided no meaningful performance boost.
  • The Noise Mismatch Problem: For more complex consumer questions, the highly specialized biomedical concepts residing within UMLS failed to align well with the broader nature of the queries. Instead of helping, the retrieved triples introduced irrelevant technical noise that actively derailed the model's reasoning.

Key Results

The authors evaluated the framework across a diverse suite of models ranging from <5B to 32B parameters using ROUGE-L and BERTScore.

Evaluation of ROUGE-L and BERTScore for base models and their KG-based RAG-enhanced variants.

As shown in the evaluation table:

  • For the majority of models (including Borea-Phi-3.5, LLaMA-3-ELYZA-JP-8B, and Gemma-3-12B-it), the addition of RAG resulted in marginal performance fluctuations of less than 1%.
  • Some models suffered noticeably; Mistral-7B experienced a performance drop of up to -3.46% in ROUGE-L when RAG was applied.
  • Phi-4-14B stood out as a major exception, demonstrating significant gains with RAG—boosting its ROUGE-L score by +4.78% on the ExpertQA-Med dataset. This suggests that certain model architectures are substantially more receptive to structured context injection than others.

Limitations and Future Work

The primary limitation lies in the alignment and translation steps. Relying on word-level translation to bridge Japanese queries with an English knowledge base can strip away critical clinical nuances. Furthermore, the structural mismatch between rigid, highly technical database triples (like those in UMLS) and colloquial consumer health questions remains a core bottleneck.

The authors suggest that future work should move away from cross-lingual pipelines and focus on constructing native, high-quality Japanese medical knowledge bases, alongside developing better methods to filter out irrelevant information before it reaches the LLM prompt.

Summary updated

Abstract

Large language models (LLMs) perform well in medical QA, but their effectiveness in Japanese contexts is limited due to privacy constraints that prevent the use of commercial models like GPT-4 in clinical settings. As a result, recent efforts focus on instruction-tuning open-source LLMs, though the potential of combining them with retrieval-augmented generation (RAG) remains underexplored. To bridge this gap, we are the first to explore a knowledge graph-based (KG) RAG framework for Japanese medical QA small-scale open-source LLMs. Experimental results show that KG-based RAG has only a limited impact on Japanese medical QA using small-scale open-source LLMs. Further case studies reveal that the effectiveness of the RAG is sensitive to the quality and relevance of the external retrieved content. These findings offer valuable insights into the challenges and potential of applying RAG in Japanese medical QA, while also serving as a reference for other low-resource languages.
BibTeX
@misc{chen2025exploring,
  author = {Yingjian Chen and Feiyang Li and Xingyu Song and Tianxiao Li and Zixin Xu and Xiujie Chen and Issey Sukeda and Irene Li},
  title = {{Exploring the Role of Knowledge Graph-Based RAG in Japanese Medical Question Answering with Small-Scale LLMs}},
  year = {2025},
  eprint = {2504.10982},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.10982},
}
Figure