Benchmarking Large-Language Models for Resource-Efficient Medical AI for Edge Deployment

Awal Ahmed Fime, Md. Zarif Hossain, Saika Zaman, Abdur Rahman Bin Shahid, Ahmed ImteajAAAI 2025 · OpenReview

TL;DR

This paper benchmarks several state-of-the-art Large Language Models (LLMs) for medical question-answering tasks, focusing on their performance and resource efficiency for deployment on edge devices. They found that Mistral v0.3 offers the best balance of accuracy and efficiency, making it a promising candidate for real-world medical applications.

What Problem Does This Paper Address?

Large Language Models (LLMs) show immense promise for revolutionizing healthcare by improving clinical workflows and assisting medical professionals. However, a significant challenge lies in deploying these powerful models in resource-constrained environments, such as edge devices (e.g., wearable health monitors). These devices have limited computational power and memory, making it difficult to run large, complex LLMs effectively. This paper aims to bridge this gap by evaluating and comparing the performance and resource efficiency of various LLMs specifically for medical applications, with an eye towards practical edge deployment.

Methods

The researchers employed Parameter-Efficient Fine-Tuning (PEFT) techniques, specifically LoRA (Low-Rank Adaptation), to adapt pre-trained LLMs for medical question-answering. PEFT methods are crucial here as they allow for fine-tuning LLMs by only updating a small subset of parameters, significantly reducing computational and memory requirements compared to full fine-tuning.

The study involved:

  • Selecting four diverse LLMs: Gemma 2 (9B), Llama 3.1 (8B), Mistral v0.3 (7B), and Phi-3.5 (mini). These were chosen for their varying sizes and architectures.
  • Fine-tuning on medical question-answering datasets: The models were trained on structured medical Q&A data to specialize them for healthcare queries.
  • Evaluating performance using BERTScore and USEScore: These metrics measure semantic similarity between the model's generated answers and ground truth, providing a robust assessment of accuracy and relevance.
  • Analyzing resource efficiency: This included examining training time and memory usage (VRAM).

Datasets

The models were trained and evaluated on five real-world medical question-answering datasets:

  • Medical Meadow Wikidoc: Features question-answer pairs from WikiDoc.
  • AI Medical Chatbot: A large dataset of over 21.2M conversational texts between doctors and patients.
  • Mental Health dataset: Sourced from online counseling platforms, containing Q&A pairs in English focused on mental health topics.
  • Medical Dataset (Jayantdocplix): Contains over 3.7k human and AI assistant conversations.
  • HealthCareMagic-100k-en: Includes over 112k medical records in English.

Key Insights and Ideas

  • Mistral v0.3 is the current champion for medical Q&A on edge: It consistently outperformed other models in both accuracy (BERTScore, USEScore) and resource efficiency.
  • Architectural innovations matter more than raw parameter count: Mistral v0.3, despite being smaller than Llama 3.1 and Gemma 2, achieved better results due to its optimized attention mechanisms and PEFT strategies.
  • PEFT is essential for edge deployment: Techniques like LoRA are critical for adapting large models without prohibitive computational costs.
  • Trade-offs exist: Smaller models like Phi-3.5 (mini) are computationally efficient but sacrifice performance on complex tasks. Larger models offer better accuracy but require more resources.
  • Dataset specificity matters: While Mistral excels on structured medical data, Llama 3.1 showed strength in conversational AI contexts.

Key Results

  • Mistral v0.3 achieved the highest F1 scores across multiple datasets, including Medical Meadow Wikidos (0.8358), Medical Dataset (0.8893), and Health Care Magic (0.745) using BERTScore.
  • Llama 3.1 showed the highest F1 score (0.8940) on the AI Medical Dataset, demonstrating its capability in specific conversational scenarios.
  • Phi-3.5 (mini) consistently showed the weakest performance, indicating its limitations for complex medical tasks.
  • Mistral v0.3 demonstrated the most efficient VRAM usage among the larger models, attributed to its PEFT approach and optimized attention layers.
  • Training time varied significantly, with Phi-3.5 (mini) being the fastest due to its smaller size, while Mistral v0.3 offered a good balance between training time and performance.

Limitations and Future Work

Limitations: * The study focused on a specific set of LLMs and medical Q&A tasks. * While resource efficiency was evaluated, a deeper dive into real-world edge device performance could be beneficial.

Future Work: * Improving model robustness and expanding the range of medical tasks. * Further optimizing architectures specifically for edge-based deployments. * Investigating the impact of different PEFT techniques and hyperparameter tuning. * Exploring multimodal LLMs for richer medical applications.

Summary updated

Abstract

Large-Language Models (LLMs) are rapidly emerging as transformative tools across diverse domains, leveraging extensive training on vast and heterogeneous datasets to capture nuanced knowledge and transcend traditional boundaries of understanding. In the medical domain, LLMs hold immense potential to revolutionize clinical workflows by enhancing the efficiency of medical practitioners and alleviating their workload. However, a critical gap exists between the theoretical capabilities of LLMs and their practical deployment in resource-constrained environments, such as edge devices (e.g., health monitors) commonly used in healthcare settings. This paper addresses this challenge by employing parameter-efficient fine-tuning (PEFT) techniques to adapt widely available advanced LLMs for medical applications while comparing their resource efficiency and performance. The models are fine-tuned on structured medical question answering datasets, and their outputs are evaluated using BERTScore and USEScore metrics. Among the models tested, Mistral v0.3 demonstrated the best performance based on both metrics, while also exhibiting promise for resource efficiency. These findings provide a vital foundation for selecting and optimizing LLMs for healthcare tasks, offering actionable insights for developing resource-efficient and scalable solutions that are well-suited for deployment on edge devices in real-world medical environments.
BibTeX
@inproceedings{fime2025benchmarking,
  author = {Awal Ahmed Fime and Md. Zarif Hossain and Saika Zaman and Abdur Rahman Bin Shahid and Ahmed Imteaj},
  title = {{Benchmarking Large-Language Models for Resource-Efficient Medical AI for Edge Deployment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=xIhQU36UNA},
}
Figure