Large Language Model Synergy for Ensemble Learning in Medical Question Answering: Design and Evaluation Study

Han Yang, Mingchen Li, Huixue Zhou, Yongkang Xiao, Qian Fang, Shuang Zhou, Rui ZhangJournal of Medical Internet Research 2025 · DOI

TL;DR

This paper introduces LLM-Synergy, a novel framework that combines multiple large language models (LLMs) to improve performance on medical question-answering (QA) tasks. By leveraging two distinct ensemble strategies, LLM-Synergy significantly outperforms individual LLMs, offering a more robust and adaptable solution for medical AI.

What Problem Does This Paper Address?

Medical question-answering (QA) is a critical area for healthcare, aiming to provide accurate and reliable information to professionals and patients. While individual large language models (LLMs) have shown promise, they often exhibit inconsistent performance across different medical QA datasets. This variability stems from their unique architectures, training data, and inherent limitations like "hallucinations" (generating incorrect information). Furthermore, identifying the single "best" LLM for every medical QA scenario is challenging due to these performance differences. The paper addresses this by proposing ensemble learning, a method that combines multiple models to harness their collective strengths and mitigate individual weaknesses, specifically for the complex domain of medical QA.

Methods

The LLM-Synergy framework employs two novel ensemble strategies:

  1. Boosting-based Weighted Majority Vote Ensemble: This method adaptively assigns weights to individual LLMs based on their performance. It iteratively refines these weights, giving more importance to models that correctly answer questions, thereby creating a more robust decision-making process. This is a "static" approach where weights are determined during training and applied consistently.
  2. Cluster-based Dynamic Model Selection Ensemble: This is a more sophisticated approach that first clusters questions based on their semantic similarity using embeddings from a specialized Clinical-BERT model. Then, for each cluster, it applies the Boosting-based Weighted Majority Vote ensemble, but with weights tailored specifically to that cluster's characteristics. This dynamic selection allows the framework to leverage the most suitable LLMs for different types of medical questions, offering a more context-aware solution.

Both methods are designed to work in a zero-shot manner, meaning they don't require extensive fine-tuning of the base LLMs, making them more efficient.

Datasets

The study evaluated its framework on three established medical QA datasets:

  • MedMCQA: A comprehensive multiple-choice question dataset derived from Indian medical entrance exams, featuring 4-option questions.
  • PubMedQA: A dataset curated from PubMed abstracts, focusing on biomedical questions that can be answered with "yes," "no," or "maybe."
  • MedQA-USMLE: A dataset of multiple-choice questions based on the United States Medical Licensing Examination, featuring 5-option questions.

The paper does not introduce new benchmarks but uses these existing ones to rigorously evaluate its proposed ensemble methods.

Key Insights and Ideas

  • LLM Diversity is Key: The paper highlights that different LLMs have distinct strengths and weaknesses, making them suitable for different types of medical questions. For example, MedAlpaca-13B performed well on MedQA-USMLE, while Llama2-13B excelled on PubMedQA.
  • Ensembling Outperforms Individuals: The core idea that combining models can lead to superior performance is strongly validated. Both ensemble methods consistently beat the best-performing individual LLM across all datasets.
  • Dynamic Selection Offers an Edge: The Cluster-based Dynamic Model Selection, by tailoring model weights to specific question clusters, provides a further performance boost over the static Boosting-based Weighted Majority Vote. This suggests that context-aware adaptation is crucial for maximizing LLM performance in complex domains.
  • Medical QA is Nuanced: The case studies demonstrate how different LLMs are better suited for different question types. Models trained on broader corpora (like MedAlpaca) might excel at fact-based questions, while those trained on clinical notes (like MedLlama) are better for nuanced patient information.

Key Results

The LLM-Synergy framework demonstrated significant improvements over individual LLMs:

  • Boosting-based Weighted Majority Vote: Achieved accuracies of 35.84% on MedMCQA (+3.81% improvement), 96.21% on PubMedQA (+0.64% improvement), and 37.26% on MedQA-USMLE.
  • Cluster-based Dynamic Model Selection: Yielded even higher accuracies of 38.01% on MedMCQA (+5.98% improvement), 96.36% on PubMedQA (+1.09% improvement), and 38.13% on MedQA-USMLE (+0.87% improvement).

Notably, the Cluster-based Dynamic Model Selection achieved the highest accuracy across all three datasets, outperforming the Boosting-based method.

Limitations and Future Work

  • LLM Dependency: The effectiveness of LLM-Synergy critically depends on the quality and diversity of the constituent LLMs. If models share similar biases, the ensemble's improvement might be limited.
  • Static Weights (for one method): The Boosting-based Weighted Majority Vote uses fixed weights, which might not adapt to question-specific nuances.
  • Clustering Algorithm Reliance: The success of the dynamic selection method relies on the clustering algorithm's ability to capture relevant question features and the availability of a sufficiently large validation set for tuning.
  • Computational Overhead: While avoiding fine-tuning, the ensemble methods still involve multiple LLM inferences and embedding computations, which can increase latency compared to single models.
  • Future Work: The authors suggest exploring increased model diversity, dynamic weighting strategies, and broader domain applications. Benchmarking against other ensemble strategies and traditional fine-tuning methods is also recommended.

Summary updated

Abstract

The LLM-Synergy framework, using 2 ensemble methods, represents a significant advancement in leveraging LLMs for medical QA tasks. Through effectively combining the strengths of diverse LLMs, this framework provides a flexible and efficient strategy adaptable to current and future challenges in biomedical informatics.
BibTeX
@article{yang2025large,
  author = {Han Yang and Mingchen Li and Huixue Zhou and Yongkang Xiao and Qian Fang and Shuang Zhou and Rui Zhang},
  title = {{Large Language Model Synergy for Ensemble Learning in Medical Question Answering: Design and Evaluation Study}},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/70080},
}
Figure