TL;DR
This paper introduces GameQA, a gamified mobile platform designed to build question-answering (QA) datasets for low-resource languages. By leveraging crowdsourcing and gamification, GameQA successfully created a large, diverse dataset for Icelandic, demonstrating a cost-effective and culturally relevant approach to QA data collection.
What Problem Does This Paper Address?
Building high-quality Question-Answering (QA) datasets, especially for languages with limited digital resources (low-resource languages), is a significant challenge. Traditional methods often rely on expensive human annotators and single, large knowledge sources like Wikipedia. This approach is problematic for low-resource languages because:
- Limited Knowledge Bases: They often lack a single, comprehensive source of information, leading to a high proportion of unanswerable questions.
- Replication Difficulty: Methods that require paid annotators are not feasible in regions with limited funding.
- Cultural Relevance: Directly translating existing datasets from high-resource languages may not capture the specific cultural, historical, or geographical nuances relevant to the target language community.
The authors aim to address these limitations by creating a more accessible, cost-effective, and culturally sensitive method for QA dataset creation.
Methods
The core of the GameQA platform is a gamified mobile application and a supporting web API. The process involves several stages:
- Question Elicitation: Users are presented with an image as a stimulus and asked to write a question. The image serves as a prompt for curiosity, encouraging users to think beyond simple descriptions.
- Question Review: Other users review the generated questions based on clarity, consistency, and answerability within a reasonable length.
- Web Search and Paragraph Selection: Users search for answers to the reviewed questions using a web search interface that accesses multiple pre-defined sources (e.g., Wikipedia, news sites, government information websites). They then select the specific paragraph containing the answer.
- Answer Span Marking: Users identify the exact span of text within the selected paragraph that answers the question. For yes/no questions, they mark the answer accordingly.
- Answer Review: A final review step ensures the identified answer span is correct.
Key Innovations:
- Gamification: To incentivize unpaid crowd-workers, GameQA incorporates game-like elements such as points, level-ups, streaks, avatars, and prestige tokens. This aims to increase user engagement and the volume of annotations.
- Social Features: Leaderboards and notifications about user contributions foster a sense of community and competition.
- Multiple Answer Sources: Unlike many existing datasets that rely on a single source, GameQA integrates multiple sources, increasing the likelihood of finding answers and creating a more diverse dataset.
- User-Centered Development: The platform was developed iteratively with user feedback, leading to a simplified interface and the inclusion of mobile accessibility.
Datasets
The primary dataset introduced and used in this paper is RUQUAD (Reykjavik University Question Answering Dataset), created for Icelandic. This dataset was built using the GameQA platform and sourced answers from five sources across four domains: Icelandic Wikipedia, Vísindavefurinn (The Icelandic Web of Science), mbl.is and visir.is (news websites), and Stjórnarráðið (The Icelandic Government Information website).
The paper also references and compares its findings with existing datasets like SQuAD, TyDi, and MS-MARCO, particularly in discussions about unanswerable question ratios and answer span lengths.
Key Insights and Ideas
- Gamification as a Motivator: The study demonstrates that gamifying the data collection process can effectively incentivize unpaid crowd-workers to contribute a significant amount of data, even without financial compensation.
- Cultural Relevance is Achievable: By allowing users to draw from local sources and use culturally relevant prompts (images), the platform naturally generates questions that reflect the local culture, history, and geography, which is crucial for building truly useful QA systems for specific communities.
- Multi-Domain Approach Benefits Low-Resource Languages: Using multiple, diverse sources, even if individually smaller than Wikipedia, is more effective for low-resource languages than relying on a single, potentially incomplete, source. This significantly reduces the unanswerable question rate.
- User-Centered Design Enhances Usability: Iterative development with user feedback led to a more intuitive and engaging platform, which is critical for sustained crowd-sourcing efforts.
Key Results
- Dataset Size: The GameQA platform successfully collected the RUQUAD dataset, which includes 23,036 questions, with 90% passing double peer review.
- Unanswerable Rate: The RUQUAD dataset achieved an unanswerable rate of 38.4%. This is notably lower than many other low-resource language datasets and even competitive with some high-resource language datasets, especially considering the limited size of Icelandic Wikipedia.
- User Engagement: 1,524 users created accounts, and 1,024 contributed content. A total of 137,972 annotation tasks were performed.
- Contribution Distribution: User contributions followed a Pareto distribution, with a minority of users performing the majority of the work.
- Answer Span Diversity: The dataset exhibits a higher variance in answer span lengths compared to SQuAD and TyDi, which is expected given the challenges of standardizing annotations across a large, distributed crowd.
Limitations and Future Work
- Image Prompts vs. Text Prompts: The paper acknowledges that the impact of using images as prompts versus traditional text prompts on question answerability and difficulty is unclear. Future work could explore this difference.
- User Responsibility for Search: While giving users freedom to find answers, the reliance on their search query skills might limit their ability to find answers if their queries are not effective. Automating this step with better search engine integration could be explored.
- Ambiguity in Questions: Some questions might be ambiguous or have multiple correct answers. The current question review step might need enhancements to handle such cases, potentially requiring question rewrites.
- Annotation Standardization: The variance in answer span lengths suggests that enforcing strict annotation standards across a large crowd is challenging. Future work could focus on improving standardization and shortening answer spans post-crowdsourcing.
The authors propose that the gamified crowdsourcing approach used in GameQA can be extended to other NLP tasks, such as machine translation, by identifying user bases interested in those domains.
Summary updated