Introduction
Medical Knowledge Graphs (KGs) are essential for clinical decision support, yet they often suffer from incompleteness. As medical coding systems like ICD and ATC evolve, many valid drug-disease relationships remain unmapped in structured databases. While Large Language Models (LLMs) are frequently proposed as a solution for "knowledge imputation"—filling these gaps using their vast training data—this study evaluates their reliability in high-stakes clinical contexts.
Methodology
The researchers evaluated four state-of-the-art models: GPT-4o, OpenAI o3-mini, Perplexity-R1, and Perplexity-Sonar. A "ground truth" benchmark was established by integrating gold-standard databases, including UMLS, DrugBank, and PrimeKG, resulting in over 100,000 known drug-disease relationships.
The evaluation utilized three distinct prompt strategies:
- Prompt A: A simple list request.
- Prompt B: A granular request for Level 5 ATC codes.
- Prompt C: A highly structured JSON format including disease descriptions.
Performance was measured using Success Rate (the ability to identify at least one correct treatment) and Recall (the percentage of all known treatments identified).
Key Results: The Precision-Recall Paradox
The study reveals a significant "precision-recall paradox." As prompts become more structured and detailed (Prompt C), models become highly successful at identifying at least one correct drug. However, this precision comes at the expense of comprehensive coverage.
As shown in the figure above, while Success Rates for Prompt C reached as high as 0.9667 for models like GPT-4o and Perplexity-Sonar, the corresponding Recall scores were dismal, ranging from 0.0074 to 0.0149. Even under the most permissive conditions (Prompt A), the highest recall achieved was only 0.0842. This indicates that while LLMs can reliably name a drug for a condition, they fail to map the full therapeutic landscape.
Discussion
The "Echo Chamber" Effect
A concerning finding was that more detailed prompts (Prompt C) led to higher agreement between different models (Dice scores up to 0.85) while simultaneously decreasing their alignment with the actual medical KG (below 0.03). This suggests that LLMs may share common biases or "reasoning shortcuts" that diverge from established clinical reality.
Platform and Structural Instability
The research also highlighted inconsistencies based on access methods. The same model (Perplexity-Sonar) yielded different results when accessed via API versus a web portal, with the latter showing significantly more inconsistency. Furthermore, structured JSON prompts forced models to be more conservative, increasing precision but further suppressing recall.
Conclusion and Future Directions
Current LLMs are not yet reliable for autonomous medical knowledge imputation. Their high success rate in finding single associations masks a fundamental inability to capture the breadth of clinical knowledge required for robust KGs.
Future work should focus on moving toward Retrieval-Augmented Generation (RAG) to ground outputs in clinical guidelines, and developing self-verification mechanisms where models check their work against medical ontologies before outputting relationships.
Summary updated