TL;DR
This paper presents the first exploration of a Knowledge Graph-based Retrieval-Augmented Generation (KG-RAG) framework tailored for Japanese medical question answering (QA) using small-scale, open-source LLMs. The authors discover that external knowledge injection provides surprisingly limited benefits, as it often introduces specialized noise or simply replicates knowledge the models already inherently possess.
What Problem Does This Paper Address?
While large language models (LLMs) excel at medical QA, deploying commercial giants like GPT-4 in real-world clinical settings is frequently blocked by strict data privacy regulations. This has driven interest in small-scale, open-source LLMs that can be hosted locally.
However, developing medical QA systems for non-English languages faces a steep hurdle: a severe scarcity of high-quality, localized medical training data. To bypass this resource constraint, researchers often turn to Retrieval-Augmented Generation (RAG) to pull in external knowledge. While text-based RAG is common, leveraging structured Knowledge Graphs (KGs) for Japanese medical QA remains largely underexplored. The authors set out to determine if anchoring small-scale LLMs to a medical knowledge graph can overcome the data scarcity problem in low-resource languages.
Methods
Because comprehensive Japanese medical knowledge graphs are difficult to access or non-existent, the authors designed a cross-lingual pipeline that leverages the massive, English-centric Unified Medical Language System (UMLS).
As illustrated in the pipeline above, the framework operates in four distinct steps:
- Entity Extraction: A base LLM extracts key medical entities from the user's Japanese query.
- Cross-Lingual Retrieval: The extracted Japanese entities undergo word-level translation into English to query the UMLS database, retrieving relevant knowledge graph triples (subject-predicate-object).
- Multi-Angle Ranking: A specialized biomedical re-ranker (
MedCPT Cross Encoder) scores and selects the top-$k$ most relevant triples based on their semantic similarity to the original question. - Declarative Conversion & Answering: The selected English triples are translated back and converted into natural Japanese declarative sentences by an LLM. These facts are appended to the original prompt as context for the final answer generation.
Datasets
The framework was evaluated on three distinct long-form medical QA datasets, originally curated in English and translated into Japanese using a sophisticated multi-stage agentic machine translation framework:
- ExpertQA-Bio: 96 high-quality biological questions verified by domain experts.
- ExpertQA-Med: 504 expert-verified medical questions.
- LiveQA: 104 test pairs consisting of consumer health questions submitted to the National Library of Medicine.
The dataset statistics are detailed below:
Key Insights and Ideas
The most striking takeaway from this study is that KG-based RAG has a highly volatile and often negligible impact on small-scale LLMs. The authors' case studies revealed a double-edged sword regarding external knowledge injection:
- The Redundancy Problem: For straightforward or general medical questions, the retrieved KG triples were accurate, but the baseline LLMs already possessed this information internally. Consequently, RAG provided no meaningful performance boost.
- The Noise Mismatch Problem: For more complex consumer questions, the highly specialized biomedical concepts residing within UMLS failed to align well with the broader nature of the queries. Instead of helping, the retrieved triples introduced irrelevant technical noise that actively derailed the model's reasoning.
Key Results
The authors evaluated the framework across a diverse suite of models ranging from <5B to 32B parameters using ROUGE-L and BERTScore.
As shown in the evaluation table:
- For the majority of models (including
Borea-Phi-3.5,LLaMA-3-ELYZA-JP-8B, andGemma-3-12B-it), the addition of RAG resulted in marginal performance fluctuations of less than 1%. - Some models suffered noticeably;
Mistral-7Bexperienced a performance drop of up to -3.46% in ROUGE-L when RAG was applied. Phi-4-14Bstood out as a major exception, demonstrating significant gains with RAG—boosting its ROUGE-L score by +4.78% on the ExpertQA-Med dataset. This suggests that certain model architectures are substantially more receptive to structured context injection than others.
Limitations and Future Work
The primary limitation lies in the alignment and translation steps. Relying on word-level translation to bridge Japanese queries with an English knowledge base can strip away critical clinical nuances. Furthermore, the structural mismatch between rigid, highly technical database triples (like those in UMLS) and colloquial consumer health questions remains a core bottleneck.
The authors suggest that future work should move away from cross-lingual pipelines and focus on constructing native, high-quality Japanese medical knowledge bases, alongside developing better methods to filter out irrelevant information before it reaches the LLM prompt.
Summary updated