TL;DR
This paper introduces a novel benchmark dataset of Polish medical licensing exams to evaluate Large Language Models (LLMs) on medical knowledge transfer, particularly across Polish and English. The study reveals that while top LLMs like GPT-40 achieve near-human performance on these exams, significant challenges remain in cross-lingual understanding and domain-specific nuances, highlighting the need for careful evaluation before clinical deployment.
What Problem Does This Paper Address?
The rapid advancement of Large Language Models (LLMs) has shown immense promise in specialized domains like medicine. However, most research and benchmarks have focused on English-language medical knowledge, leaving a significant gap in understanding LLM performance in other linguistic contexts. This is particularly problematic because medical practices and guidelines can vary across countries, and LLMs trained predominantly on English data might exhibit biases or fail to capture specific regional medical knowledge. This paper addresses this by creating a benchmark and evaluating LLMs on Polish medical licensing exams, aiming to assess their capabilities in a non-English medical domain and explore cross-lingual knowledge transfer.
Methods
The authors developed a new benchmark dataset comprising over 22,000 questions from Polish medical licensing and specialization exams (LEK, LDEK, PES). This dataset includes both Polish original questions and professionally translated English versions for comparative analysis. They evaluated a range of LLMs, categorized into medical-specific, general-purpose multilingual, and Polish-specific models, as well as models with restricted APIs. The evaluation involved directly prompting these models with exam questions, mimicking the actual exam environment without few-shot prompting. Performance was measured by the percentage of correct answers and the ability to pass exams based on established thresholds.
Datasets
- New Benchmark Dataset: The core contribution is a novel dataset derived from publicly available Polish medical licensing and specialization exams (LEK, LDEK, PES). This dataset includes over 22,000 questions spanning various medical specialties.
- Sub-datasets: The benchmark is further divided into:
- LEK (Polish)
- LDEK (Polish)
- PES (Polish)
- LEK (en) - English translation of LEK questions
- LDEK (en) - English translation of LDEK questions
- Existing Datasets (for context): The paper references existing medical QA benchmarks like MedQA, MedMCQA, and PubMedQA, noting their English-centric nature.
Key Insights and Ideas
- Cross-lingual Transfer is Real, But Imperfect: LLMs demonstrate a significant advantage when tested on English versions of medical exams compared to their Polish counterparts, even when the models are Polish-specific. This suggests that while LLMs can transfer knowledge across languages, the depth of understanding in non-English domains is still developing.
- General-Purpose LLMs Outperform Domain-Specific (in this context): Surprisingly, general-purpose multilingual LLMs often outperformed models specifically fine-tuned on English medical data. This might be due to the sheer scale and diversity of the training data for general models.
- GPT-40 is a Strong Performer, But Not Perfect: The GPT-40 model stands out as the top performer, achieving near-human levels of accuracy and passing most exams. However, it still struggles with certain PES specializations, indicating that even state-of-the-art models have limitations.
- Model Size Matters for Cross-Lingual Performance: As LLM size increases, the performance gap between Polish and English versions of the exams tends to narrow, suggesting that larger models are better equipped for robust cross-lingual understanding.
- Human Performance Remains the Gold Standard: While LLMs are improving, they still lag behind average human medical students and doctors in many areas, particularly in complex PES specializations.
Key Results
- GPT-40 Achieves Near-Human Performance: GPT-40 achieved the highest overall accuracy, scoring 89.40% on LEK, 75.63% on LDEK, and 75.35% on PES. It passed all LEK and LDEK exams and most PES exams.
- English Advantage: Most models performed significantly better on the English versions of the LEK and LDEK exams compared to their Polish counterparts. For example, Meta-Llama-3.1-8B-Instruct passed all 13 LEK exams in English but only one in Polish.
- General-Purpose Models Lead: Top-performing open-source models like Meta-Llama-3.1-70B-Instruct outperformed medical-specific models.
- Specialty Performance Varies: LLM performance varied greatly across different medical specialties, with dentistry (orthodontics, conservative dentistry) being particularly challenging. Laboratory diagnostics and public health were areas where LLMs performed better.
- Comparison to Humans: Top LLMs like GPT-40 and Meta-Llama-3.1-70B-Instruct performed comparably to average medical students on LEK exams, but struggled more on LDEK and PES exams, often scoring below the median human performance.
Limitations and Future Work
- Limited Scope: The benchmark focuses on multiple-choice exams, which represent only a fraction of a physician's actual clinical skills. Real-world medical practice involves complex problem-solving, patient interaction, and diverse data analysis, which are not fully captured by these exams.
- Data Availability: Certain models (e.g., PaLM 2, some Llama 3.2 variants, Gemini) could not be evaluated due to access restrictions.
- Potential for Training Data Contamination: The authors acknowledge the risk that evaluated LLMs might have been trained on the benchmark data, a common issue with publicly available datasets.
- Future Work: The authors suggest that their benchmark can be used to investigate cross-lingual transfer more deeply and to identify LLM weaknesses where language versions differ. They also highlight the need for more comprehensive evaluations that go beyond exam performance to assess real-world clinical utility.
Summary updated