Introduction
Traditional Question Answering (QA) systems have achieved significant success in addressing "factoid" questions—queries with concise, objective answers like dates or names. However, "Why" questions present a greater challenge, requiring longer, discourse-based explanations that are often subjective or context-dependent. The primary bottleneck in advancing systems for these complex queries is the lack of large-scale, high-quality evaluation datasets. Manual creation of such corpora is prohibitively expensive and slow. Researchers at the Tokyo Institute of Technology addressed this by developing the WhyQA-corpus, a specialized dataset of Why-questions and answers derived from Wikipedia, using a cost-effective crowdsourcing pipeline.
Methodology
The researchers designed a three-phase workflow on Amazon Mechanical Turk (MTurk) to distribute cognitive load and ensure data quality. The pipeline consists of specialized Human Intelligence Tasks (HITs) designed to move from question generation to answer selection and linguistic variation.
- Question Generation (QGenHIT): Workers were shown partial Wikipedia snippets (titles, headers, and introductory sentences) and asked to write a "Why" question answerable by the full article. This "partial content" approach encouraged natural curiosity over simple text copying.
- Answering (QAHIT): Different workers were presented with the generated questions and the full articles. They selected 1–3 sentences as answers. To capture a diversity of perspectives, each question was assigned to 8–10 different workers.
- Rephrasing (QRepHIT): To increase linguistic variety, workers rephrased original questions into five distinct versions.
Quality control was maintained through a mandatory qualification test and monitoring of "Time On Task" to filter low-effort submissions.
The WhyQA-Corpus
The resulting WhyQA-corpus is based on 84 Wikipedia articles and contains 695 questions (536 answerable and 159 unanswerable). For answerable questions, the corpus provides 8–10 overlapping human-selected answers and five rephrased question variants. The dataset is structured in XML and released under the GNU Free Documentation License, providing a rich resource for training and evaluating non-factoid QA systems.
Key Results and Insights
The study demonstrated that high-quality, non-factoid QA data can be collected quickly and inexpensively. The entire corpus was built for approximately $350, a fraction of the cost of expert-led collection.
A critical finding was the high level of worker consensus. Using the ROUGE metric, the authors measured agreement among workers for answerable questions. As shown in the results, the crowd reached a meaningful consensus that significantly outperformed random chance.
The ROUGE-1 score of 0.56 for worker agreement (when "No Answer" (NoA) is not included) is substantially higher than the 0.13 score for random answers. This supports the authors' argument that the "ground truth" for complex questions can be effectively defined by the consensus of multiple human perspectives.
Conclusion
The WhyQA-corpus project proves that crowdsourcing can generate reliable, complex QA data if the workflow is properly structured. While some low-quality work required manual rejection, the overall pipeline was highly efficient. Future work will focus on using answer agreement data to develop automated quality control tools and testing how rephrased questions impact the performance of automated QA models. This framework offers a scalable path toward building the large-scale datasets necessary for the next generation of conversational and explanatory AI.
Summary updated