TL;DR
This paper benchmarks several state-of-the-art Large Language Models (LLMs) for medical question-answering tasks, focusing on their performance and resource efficiency for deployment on edge devices. They found that Mistral v0.3 offers the best balance of accuracy and efficiency, making it a promising candidate for real-world medical applications.
What Problem Does This Paper Address?
Large Language Models (LLMs) show immense promise for revolutionizing healthcare by improving clinical workflows and assisting medical professionals. However, a significant challenge lies in deploying these powerful models in resource-constrained environments, such as edge devices (e.g., wearable health monitors). These devices have limited computational power and memory, making it difficult to run large, complex LLMs effectively. This paper aims to bridge this gap by evaluating and comparing the performance and resource efficiency of various LLMs specifically for medical applications, with an eye towards practical edge deployment.
Methods
The researchers employed Parameter-Efficient Fine-Tuning (PEFT) techniques, specifically LoRA (Low-Rank Adaptation), to adapt pre-trained LLMs for medical question-answering. PEFT methods are crucial here as they allow for fine-tuning LLMs by only updating a small subset of parameters, significantly reducing computational and memory requirements compared to full fine-tuning.
The study involved:
- Selecting four diverse LLMs: Gemma 2 (9B), Llama 3.1 (8B), Mistral v0.3 (7B), and Phi-3.5 (mini). These were chosen for their varying sizes and architectures.
- Fine-tuning on medical question-answering datasets: The models were trained on structured medical Q&A data to specialize them for healthcare queries.
- Evaluating performance using BERTScore and USEScore: These metrics measure semantic similarity between the model's generated answers and ground truth, providing a robust assessment of accuracy and relevance.
- Analyzing resource efficiency: This included examining training time and memory usage (VRAM).
Datasets
The models were trained and evaluated on five real-world medical question-answering datasets:
- Medical Meadow Wikidoc: Features question-answer pairs from WikiDoc.
- AI Medical Chatbot: A large dataset of over 21.2M conversational texts between doctors and patients.
- Mental Health dataset: Sourced from online counseling platforms, containing Q&A pairs in English focused on mental health topics.
- Medical Dataset (Jayantdocplix): Contains over 3.7k human and AI assistant conversations.
- HealthCareMagic-100k-en: Includes over 112k medical records in English.
Key Insights and Ideas
- Mistral v0.3 is the current champion for medical Q&A on edge: It consistently outperformed other models in both accuracy (BERTScore, USEScore) and resource efficiency.
- Architectural innovations matter more than raw parameter count: Mistral v0.3, despite being smaller than Llama 3.1 and Gemma 2, achieved better results due to its optimized attention mechanisms and PEFT strategies.
- PEFT is essential for edge deployment: Techniques like LoRA are critical for adapting large models without prohibitive computational costs.
- Trade-offs exist: Smaller models like Phi-3.5 (mini) are computationally efficient but sacrifice performance on complex tasks. Larger models offer better accuracy but require more resources.
- Dataset specificity matters: While Mistral excels on structured medical data, Llama 3.1 showed strength in conversational AI contexts.
Key Results
- Mistral v0.3 achieved the highest F1 scores across multiple datasets, including Medical Meadow Wikidos (0.8358), Medical Dataset (0.8893), and Health Care Magic (0.745) using BERTScore.
- Llama 3.1 showed the highest F1 score (0.8940) on the AI Medical Dataset, demonstrating its capability in specific conversational scenarios.
- Phi-3.5 (mini) consistently showed the weakest performance, indicating its limitations for complex medical tasks.
- Mistral v0.3 demonstrated the most efficient VRAM usage among the larger models, attributed to its PEFT approach and optimized attention layers.
- Training time varied significantly, with Phi-3.5 (mini) being the fastest due to its smaller size, while Mistral v0.3 offered a good balance between training time and performance.
Limitations and Future Work
Limitations: * The study focused on a specific set of LLMs and medical Q&A tasks. * While resource efficiency was evaluated, a deeper dive into real-world edge device performance could be beneficial.
Future Work: * Improving model robustness and expanding the range of medical tasks. * Further optimizing architectures specifically for edge-based deployments. * Investigating the impact of different PEFT techniques and hyperparameter tuning. * Exploring multimodal LLMs for richer medical applications.
Summary updated