TL;DR
This paper introduces TraSe, a novel Retrieval-Augmented Generation (RAG) architecture designed to improve question-answering for Bangla, a low-resource language. By combining a "Translative Prompting" method with a selection mechanism, TraSe significantly boosts accuracy, demonstrating the potential for RAG in underserved linguistic communities.
What Problem Does This Paper Address?
Natural Language Processing (NLP) has seen remarkable progress, with Retrieval-Augmented Generation (RAG) emerging as a powerful technique to enhance large language models (LLMs) by grounding their responses in external knowledge. This mitigates issues like hallucination and outdated information. However, RAG research has largely focused on high-resource languages like English, leaving low-resource languages, such as Bangla, significantly underserved. Bangla, spoken by over 230 million people, presents unique challenges for current NLP models, which struggle with accurate retrieval and contextually relevant generation. This gap limits the practical application of advanced NLP for a substantial global population.
Methods
The core of this work lies in the TraSe architecture and its accompanying Translative Prompting method.
- Translative Prompting: Recognizing that LLMs like Llama 2 have limited training data for Bangla, this method acts as a bridge. It instructs the model to translate the input query and context into English, perform the retrieval and answer generation in English, and then translate the final answer back into Bangla. This approach is particularly effective for text-based answers.
- TraSe Architecture: This architecture builds upon Translative Prompting by introducing a selection mechanism. It generates candidate answers using Translative Prompting and also from other baseline methods (like zero-shot, two-shot, Self-Ask, and ReAct). The TraSe architecture then uses a selector to choose the best answer based on the query and context. This allows it to leverage the strengths of different prompting strategies.
- Embeddings: For embedding queries and documents, the paper utilizes BanglaBERT and bert-base-multilingual-cased models. Cosine similarity is employed for retrieving the top 3 relevant contexts.
Datasets
- Bangla Wikipedia Dump: The authors created a new dataset for their experiments by extracting 200 question-answer pairs from the Bangla Wikipedia dump. This dataset is divided into 27 articles, with each question accompanied by 3 relevant contexts for human-in-the-loop (HIL) evaluation. The dataset includes both text-based and number-based answers.
Key Insights and Ideas
- Translative Prompting as a Low-Resource Bridge: The core insight is that by leveraging the LLM's stronger capabilities in English, we can effectively overcome the limitations of low-resource language training data for tasks like RAG.
- Selection Mechanism Enhances Robustness: The TraSe architecture's ability to select the best answer from multiple candidates (generated by different methods) proves more robust than relying on a single method. This highlights the benefit of ensemble-like approaches in RAG.
- Human-in-the-Loop (HIL) Significantly Boosts Performance: The results strongly suggest that human oversight or validation in the retrieval process dramatically improves RAG performance, especially for low-resource languages.
Key Results
The TraSe architecture demonstrated significant improvements in accuracy:
- Automatic Retrieval: TraSe achieved 34% accuracy with automatic retrieval.
- Human-in-the-Loop (HIL) Retrieval: TraSe achieved an impressive 63% accuracy with HIL retrieval.
- Translative Prompting Alone: For text-based answers, Translative Prompting achieved an accuracy of 0.61 with HIL context, outperforming baseline methods.
- Comparison: The paper shows that TraSe with zero-shot and Translative prompting achieved 33% accuracy with Bert-base-multilingual-case, a substantial jump from the baseline 22% accuracy. In HIL retrieval, TraSe achieved 63% accuracy compared to the baseline 51%.
Limitations and Future Work
- Limitations:
- The study used only a single language model (Llama 2 7B), which might not represent the full spectrum of LLM performance.
- The relatively small dataset size (200 Q&A pairs) could affect the generalizability of the findings.
- The focus was solely on Bangla, and results might differ for other low-resource languages.
- Future Work:
- Incorporate a variety of language models and larger, more diverse datasets to validate and generalize the findings.
- Expand the research to include other low-resource languages to gain a more comprehensive understanding of RAG performance across different linguistic contexts.
- Investigate how language models perform in diverse linguistic contexts.
Summary updated