Abstract
While Large Language Models (LLMs) have demonstrated significant potential in the medical domain, current benchmarks often treat medical knowledge as a monolithic entity. However, clinical practice is inherently specialized, with distinct terminologies and priorities across different fields. This paper introduces S-MedQA, a large-scale medical QA dataset featuring fine-grained clinical specialty annotations. Through extensive experimentation, the authors challenge the common assumption of "knowledge injection" during fine-tuning, revealing instead that specialty-specific training acts as a general domain shift that enhances performance across unrelated medical fields.
Introduction
In the rapidly evolving landscape of medical AI, a critical question remains: how does specialized clinical data influence model performance? Most existing open-source medical datasets, such as MedQA, lack the specialty-specific labels necessary to evaluate niche clinical performance. Furthermore, there is a prevailing belief that fine-tuning a model on a specific specialty "teaches" it new facts about that field. This study investigates these assumptions by developing a robust pipeline for specialty categorization and evaluating the cross-specialty transfer effects of fine-tuning.
Methodology and Dataset Construction
To address the lack of specialized data, the researchers developed S-MedQA, derived from the MedQA and MedMCQA datasets. The construction process involved several sophisticated steps to ensure high-quality annotations.
S-MedQA Pipeline
The authors utilized GPT-3.5 with five distinct prompts to categorize over 24,000 QA pairs into 15 clinical specialties. A majority voting mechanism was employed to ensure label reliability. Recognizing that many medical cases are cross-disciplinary (e.g., a patient with both cardiac and renal issues), they implemented a multi-label conformal prediction procedure. This allowed for the assignment of multiple specialty tags with statistical coverage guarantees.
Classifier-Based Evaluation
A significant technical contribution of this work is the move away from traditional "first-token probability" metrics. The authors argue that relying on the probability of the first token (e.g., 'A', 'B', 'C') suffers from selection bias. Instead, they trained a dedicated Mistral-v0.2 classifier to evaluate the model's full text output, determining if the generated explanation and final choice correctly match the ground truth. This approach provides a more accurate reflection of the model's reasoning capabilities.
Key Insights: The "Off-Diagonal" Surprise
The most striking finding of the study is the "off-diagonal" performance gain. In a standard evaluation matrix where rows represent training specialties and columns represent test specialties, one would expect the highest accuracy on the diagonal (e.g., Cardiology training helping most on Cardiology tests).
However, the results showed that the best performance often occurred off-diagonal. For example, fine-tuning Llama-3.1-8B on Neurology data frequently resulted in higher Cardiology test scores than fine-tuning on Cardiology data itself. This suggests that fine-tuning does not primarily function as "knowledge injection" of new facts. Instead, it serves as a domain shift, nudging the model from a general-purpose language space into a specialized "medical mode." Once in this mode, the model can more effectively retrieve the vast medical knowledge it already acquired during its massive pre-training phase.
Results and Discussion
Performance and Validation
Human expert validation confirmed the high quality of the S-MedQA dataset, with annotation accuracy reaching up to 97.8% and a Jaccard Similarity of 0.82 for multi-label cases. Fine-tuning on any medical specialty consistently improved performance over base models; for Llama-3.1-8B, average accuracy across all specialties rose from 65.5% to 75.1%.
The "Rising Tide" Effect
The authors observed a "rising tide" phenomenon: fine-tuning on one specialty increased the log-probabilities of clinical terms across all specialties. To ensure this wasn't due to simple terminology overlap, they used the SNOMED-CT knowledge graph to show that specialties share only about 32.8% of their unique terms. This confirms that the cross-specialty gains are a result of a broader domain alignment rather than overlapping content.
Negative Control
To further validate that these gains were specifically medical, the researchers fine-tuned models on unrelated MMLU social science data. In these cases, medical token probabilities dropped significantly, proving that the performance jumps seen with S-MedQA were uniquely tied to the medical nature of the training data.
Conclusion and Future Work
This research provides a new perspective on how LLMs process specialized knowledge. By demonstrating that "Neuro helps Cardio," the authors highlight that the value of specialized data lies in its ability to align models with the broader clinical domain.
Future work will aim to address the "long tail" of rarer clinical fields and expand the dataset beyond English-centric, US-based medical exams. Additionally, further investigation is needed to understand the specific structural similarities in reasoning that allow certain unrelated specialties to provide such high levels of transfer performance.
Summary updated