What Does Neuro Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs

Xinlan Yan, Di Wu, Yibin Lei, Christof Monz, Iacer CalixtoarXiv 2025 · DOI · arXiv

Abstract

While Large Language Models (LLMs) have demonstrated significant potential in the medical domain, current benchmarks often treat medical knowledge as a monolithic entity. However, clinical practice is inherently specialized, with distinct terminologies and priorities across different fields. This paper introduces S-MedQA, a large-scale medical QA dataset featuring fine-grained clinical specialty annotations. Through extensive experimentation, the authors challenge the common assumption of "knowledge injection" during fine-tuning, revealing instead that specialty-specific training acts as a general domain shift that enhances performance across unrelated medical fields.

Introduction

In the rapidly evolving landscape of medical AI, a critical question remains: how does specialized clinical data influence model performance? Most existing open-source medical datasets, such as MedQA, lack the specialty-specific labels necessary to evaluate niche clinical performance. Furthermore, there is a prevailing belief that fine-tuning a model on a specific specialty "teaches" it new facts about that field. This study investigates these assumptions by developing a robust pipeline for specialty categorization and evaluating the cross-specialty transfer effects of fine-tuning.

Methodology and Dataset Construction

To address the lack of specialized data, the researchers developed S-MedQA, derived from the MedQA and MedMCQA datasets. The construction process involved several sophisticated steps to ensure high-quality annotations.

S-MedQA Pipeline

The authors utilized GPT-3.5 with five distinct prompts to categorize over 24,000 QA pairs into 15 clinical specialties. A majority voting mechanism was employed to ensure label reliability. Recognizing that many medical cases are cross-disciplinary (e.g., a patient with both cardiac and renal issues), they implemented a multi-label conformal prediction procedure. This allowed for the assignment of multiple specialty tags with statistical coverage guarantees.

Overview of the S-MedQA construction pipeline and manual validation process.

Classifier-Based Evaluation

A significant technical contribution of this work is the move away from traditional "first-token probability" metrics. The authors argue that relying on the probability of the first token (e.g., 'A', 'B', 'C') suffers from selection bias. Instead, they trained a dedicated Mistral-v0.2 classifier to evaluate the model's full text output, determining if the generated explanation and final choice correctly match the ground truth. This approach provides a more accurate reflection of the model's reasoning capabilities.

Comparison of evaluation methods: Classifier-based evaluation vs. first-token probability.

Key Insights: The "Off-Diagonal" Surprise

The most striking finding of the study is the "off-diagonal" performance gain. In a standard evaluation matrix where rows represent training specialties and columns represent test specialties, one would expect the highest accuracy on the diagonal (e.g., Cardiology training helping most on Cardiology tests).

However, the results showed that the best performance often occurred off-diagonal. For example, fine-tuning Llama-3.1-8B on Neurology data frequently resulted in higher Cardiology test scores than fine-tuning on Cardiology data itself. This suggests that fine-tuning does not primarily function as "knowledge injection" of new facts. Instead, it serves as a domain shift, nudging the model from a general-purpose language space into a specialized "medical mode." Once in this mode, the model can more effectively retrieve the vast medical knowledge it already acquired during its massive pre-training phase.

Results and Discussion

Performance and Validation

Human expert validation confirmed the high quality of the S-MedQA dataset, with annotation accuracy reaching up to 97.8% and a Jaccard Similarity of 0.82 for multi-label cases. Fine-tuning on any medical specialty consistently improved performance over base models; for Llama-3.1-8B, average accuracy across all specialties rose from 65.5% to 75.1%.

The "Rising Tide" Effect

The authors observed a "rising tide" phenomenon: fine-tuning on one specialty increased the log-probabilities of clinical terms across all specialties. To ensure this wasn't due to simple terminology overlap, they used the SNOMED-CT knowledge graph to show that specialties share only about 32.8% of their unique terms. This confirms that the cross-specialty gains are a result of a broader domain alignment rather than overlapping content.

Negative Control

To further validate that these gains were specifically medical, the researchers fine-tuned models on unrelated MMLU social science data. In these cases, medical token probabilities dropped significantly, proving that the performance jumps seen with S-MedQA were uniquely tied to the medical nature of the training data.

Conclusion and Future Work

This research provides a new perspective on how LLMs process specialized knowledge. By demonstrating that "Neuro helps Cardio," the authors highlight that the value of specialized data lies in its ability to align models with the broader clinical domain.

Future work will aim to address the "long tail" of rarer clinical fields and expand the dataset beyond English-centric, US-based medical exams. Additionally, further investigation is needed to understand the specific structural similarities in reasoning that allow certain unrelated specialties to provide such high levels of transfer performance.

Summary updated

Abstract

In this paper, we introduce S-MedQA, an English medical question-answering (QA) dataset designed for benchmarking large language models (LLMs) in fine-grained clinical specialties. S-MedQA consists of over 24k examples, covering 15 medical specialties, with QA pairs that can have multiple specialty annotations, such as when a question is cross-disciplinary. The dataset is constructed using both machine and expert verification to maximize data availability and reliability. We use S-MedQA to investigate the role of clinical specialties in the knowledge-intensive scenario of medical QA. Our results show that training on data from a clinical specialty does not necessarily lead to the best performance on that specialty. Additionally, regardless of the specialty the LLM was fine-tuned on, token probabilities of clinically relevant terms consistently increase across all specialties. Based on these findings, we hypothesize that improvement gains, at least in our settings, are derived primarily from domain shifting (e.g., general to medical) rather than from injecting specialty-specific knowledge. This suggests a need to rethink the role of fine-tuning data in the medical domain. To encourage further advancements in the clinical NLP field, we release S-MedQA along with all the code required to reproduce our experiments for the research community.
BibTeX
@misc{yan2025what,
  author = {Xinlan Yan and Di Wu and Yibin Lei and Christof Monz and Iacer Calixto},
  title = {{What Does Neuro Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs}},
  year = {2025},
  eprint = {2505.10113},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.10113},
}
Figure