TL;DR
This paper introduces Y-NQ, a novel English-Yorùbá dataset for open-book reading comprehension with open-ended questions. It reveals a significant performance gap between English and Yorùbá for current large language models, suggesting their generalization capabilities do not extend to low-resource languages like Yorùbá.
What Problem Does This Paper Address?
The paper tackles the challenge of evaluating how well large language models (LLMs) can perform reading comprehension tasks, specifically with open-ended questions, across languages with vastly different resource levels. While English is a high-resource language with abundant data and advanced NLP tools, Yorùbá is a low-resource language with limited digital resources. Existing benchmarks often focus on high-resource languages, leaving a gap in understanding LLM performance and generalization for languages like Yorùbá. This research aims to bridge that gap by creating a comparable dataset and evaluating state-of-the-art LLMs on their ability to understand and generate answers in both languages.
Methods
The core of this work is the creation of the Y-NQ dataset, which is an extension of the existing Natural Questions (NQ) dataset. The authors adapted NQ's open-book reading comprehension format, which involves providing a document context and asking questions that require free-text answers. They specifically focused on creating parallel documents and questions for both English and Yorùbá. To ensure comparability, they filtered NQ for documents with specific paragraph structures and then found corresponding Yorùbá Wikipedia pages. Human annotators, proficient in both English and Yorùbá, were employed to create and validate the question-answer pairs. The authors also attempted an automatic pre-annotation step using SONAR embeddings to speed up the process, but found it unreliable due to low similarity scores.
Datasets
- Y-NQ (Yorùbá Natural Questions): This is the primary contribution of the paper. It's an English-Yorùbá evaluation dataset for open-book reading comprehension with open-ended questions. It comprises 358 questions and answers derived from 338 English documents and 208 Yorùbá documents. The dataset is designed to be comparable in topic and domain, though document and answer lengths differ.
- Natural Questions (NQ): The Y-NQ dataset is sourced from NQ, a large-scale open-book question-answering dataset.
- Wikipedia: Both English and Yorùbá documents used in the dataset are sourced from Wikipedia.
Key Insights and Ideas
- Significant Performance Disparity: LLMs exhibit a consistent and substantial performance gap between English and Yorùbá, even when Yorùbá documents are shorter. This suggests that current LLMs struggle to generalize their reading comprehension abilities to low-resource languages.
- Document Length Matters Differently: While longer documents generally pose a challenge, Yorùbá performance drops dramatically around 1500 words, whereas English performance is less affected at this length. This highlights a specific weakness in handling extended contexts in the low-resource language.
- Wikipedia Accuracy Issues: The human annotation process revealed inaccuracies in some English Wikipedia articles, even for topics that have corresponding Yorùbá content. This underscores the need for better cross-lingual linking and content validation.
- "Easier" Task, Worse Results: Counterintuitively, the Yorùbá task is considered "easier" due to shorter documents. The fact that models perform worse despite this suggests that factors beyond just context length are at play, potentially including the model's internal knowledge bias towards high-resource languages.
Key Results
- Automatic Metrics: Across all evaluated LLMs (GPT-4, 01-mini, LLaMA-3.1-8b), Yorùbá consistently scored lower than English on Rouge metrics (e.g., Rouge-1, Rouge-2, Rouge-L). For instance, GPT-4 showed a drop of 0.4 in Rouge-1 scores for Yorùbá compared to English.
- Human Evaluation: For a small subset of comparable documents, human evaluation showed English responses scoring an average of 2.33 (on a 1-3 Likert scale), while Yorùbá responses scored an average of 2.00. This qualitative assessment further supports the quantitative findings.
- Length Analysis: Figure 1 illustrates a clear drop in performance for Yorùbá when document lengths exceed approximately 1500 words, a trend less pronounced in English.
Limitations and Future Work
- Dataset Size and Comparability: The authors acknowledge that Y-NQ is limited in size, language, and domain coverage. The dataset is not fully comparable between English and Yorùbá due to differences in document and answer lengths. The use of Wikipedia might also lead to contamination issues, potentially inflating results.
- Model and Metric Limitations: The experimentation was limited to specific LLMs and automatic evaluation metrics, though compensated by a small human evaluation.
- Future Work: The paper suggests that Y-NQ can be used to evaluate how reading comprehension capabilities extend to Yorùbá and other low-resource languages. Further research could explore more diverse domains and languages, as well as develop more robust evaluation methodologies for low-resource settings. The findings also point to the need for better cross-lingual Wikipedia linking and content quality improvements.
Summary updated