TL;DR
This paper evaluates nine leading Large Language Models (LLMs) on their ability to summarize Bengali consumer health queries (CHQs) in a zero-shot setting. The study demonstrates that even without task-specific fine-tuning, LLMs like Mixtral-8x22b-Instruct can rival or even surpass the performance of a fine-tuned state-of-the-art model (Bangla T5), highlighting the growing potential of LLMs for low-resource languages in healthcare.
What Problem Does This Paper Address?
Consumer health queries (CHQs) are increasingly common, but they often contain a lot of extraneous information, making it difficult for healthcare professionals to quickly extract the essential details. Abstractive summarization offers a solution by creating concise versions of these queries. However, for low-resource languages like Bengali, there's a significant lack of annotated datasets and fine-tuned models for such tasks. This makes it challenging to leverage advanced NLP techniques. This paper aims to bridge this gap by evaluating the zero-shot summarization capabilities of modern LLMs on Bengali CHQs, assessing their potential to provide effective solutions without extensive task-specific training.
Methods
The researchers employed a zero-shot learning framework to evaluate the performance of nine LLMs. This means the models were asked to perform the summarization task without any prior training on Bengali CHQ summarization data. The evaluation was conducted using the BanglaCHQ-Summ dataset. The prompt was carefully designed and iterated upon to ensure concise, Bangla-only summaries focusing on critical information, addressing issues like models responding in English or providing overly verbose outputs. Performance was measured using standard ROUGE metrics (ROUGE-1, ROUGE-2, and ROUGE-L) and compared against Bangla T5, a fine-tuned state-of-the-art model for Bengali CHQ summarization.
Datasets
The primary dataset used in this study is BanglaCHQ-Summ. This is described as the first publicly available dataset specifically designed for consumer health query (CHQ) summarization in Bengali. It contains 2,350 annotated query-summary pairs, sourced from a popular Bengali health forum. The dataset covers 32 different health categories.
Key Insights and Ideas
- LLMs are surprisingly capable in low-resource settings: The study shows that even without specific fine-tuning, advanced LLMs can perform remarkably well on a specialized task like Bengali CHQ summarization, approaching or exceeding the performance of a fine-tuned model.
- Zero-shot learning is a viable path for low-resource languages: This work demonstrates that LLMs can be a powerful tool for languages with limited data, reducing the need for costly and time-consuming data annotation and model training.
- Mixtral-8x22b-Instruct shines: This model emerged as a top performer, particularly in ROUGE-1 and ROUGE-L metrics, indicating its strength in capturing key information and overall summary coherence.
- Bangla T5 still holds an edge in bi-gram accuracy: While LLMs excel in overall summary quality, the fine-tuned Bangla T5 model performed best on ROUGE-2, suggesting its specialized training helps in generating more accurate word sequences.
- Pronunciation variability can impact performance: The paper identifies phonetic variations in Bengali as a potential factor affecting LLM performance, suggesting this is an area for future research.
- Summary length matters: There's an interesting observation that models producing summaries closer in length to the gold standard (e.g., Llama-3-70B-Instruct) can achieve high quality, even with fewer words.
Key Results
- Mixtral-8x22b-Instruct achieved the highest scores in ROUGE-1 (51.36) and ROUGE-L (49.17).
- Bangla T5 led in ROUGE-2 (29.11).
- Models like GPT-4 and Llama-3-70B-Instruct also performed competitively across all metrics.
- Athene-70B and Qwen2-72b-Instruct showed lower performance compared to other LLMs.
- GPT-4 and Qwen2-72b-Instruct tended to generate longer summaries (around 37 words) compared to the gold standard (26 words).
- Llama-3-70B-Instruct produced significantly shorter summaries (19 words) while maintaining high performance, indicating efficiency.
Limitations and Future Work
- Pronunciation Variability: The authors note that pronunciation mismatches in Bengali can lead to score discrepancies and suggest this is a critical factor affecting LLM performance that warrants further research.
- Generalization to other domains: While this study focuses on health queries, further research is needed to explore LLM performance across other specialized domains in Bengali.
- Bridging the gap: Although LLMs are closing the gap with fine-tuned models, there's still a journey ahead to fully match their performance, especially in nuanced aspects like bi-gram accuracy. Future work should focus on improving LLM adaptability and efficiency for resource-constrained settings.
Summary updated