A Benchmark Suite of Japanese Natural Questions

Takuya Uematsu, Hao Wang, Daisuke Kawahara, Tomohide ShibataNAACL 2024

TL;DR

This paper introduces two new Japanese Question Answering (QA) datasets, JNQ and JBoolQ, designed to mirror natural human information needs by leveraging search engine query logs. These datasets aim to bridge the gap in high-quality, natural QA resources for Japanese, enabling more robust NLP model development and evaluation in the language.

What Problem Does This Paper Address?

Developing high-performance Natural Language Processing (NLP) models, particularly for Question Answering (QA), heavily relies on diverse and robust datasets for training, evaluation, and analysis. While English boasts numerous QA datasets, other languages, including Japanese, suffer from a scarcity of such resources. Existing Japanese QA datasets are often basic or suffer from "annotation artifacts" – questions manually crafted by annotators that don't reflect genuine user queries. This lack of natural, real-world questions hinders the development of NLP models that can truly understand and respond to human information needs in Japanese.

Methods

The core of this work lies in the creation of two novel Japanese QA datasets: Japanese Natural Questions (JNQ) and Japanese BoolQ (JBoolQ).

  • JNQ: This dataset is a Japanese adaptation of the English Natural Questions (NQ) dataset. Instead of direct translation (which can introduce unnaturalness due to linguistic and cultural differences), JNQ questions are collected from Japanese search engine query logs. The process involves:
  • Question and Document Collection: Queries from search logs are filtered to identify potential questions, and corresponding Japanese Wikipedia articles are retrieved as documents.
  • Good Question Identification: Crowdworkers filter out non-questions and ambiguous queries.
  • Long Answer Identification: Crowdworkers identify paragraphs within the document that contain sufficient information to answer the question.
  • Short Answer Identification: For non-yes/no questions, crowdworkers extract the shortest possible answer span from the identified long answer paragraphs.
  • JBoolQ: This dataset is a Japanese version of BoolQ, focusing on yes/no questions. It follows a similar collection process to JNQ but with a key enhancement: it includes an "unable to answer" (NONE) option, making it more challenging than the original BoolQ.

The paper also defines three specific QA tasks: long answer extraction, short answer extraction, and yes/no answer identification, and establishes baseline performance using competitive NLP models.

Datasets

  • Japanese Natural Questions (JNQ): Contains 16,871 questions and 80,288 paragraphs.
  • Japanese BoolQ (JBoolQ): Contains 6,467 questions and 31,677 paragraphs.

These datasets are derived from Japanese search engine query logs and Japanese Wikipedia articles. They are designed to be publicly available to facilitate research.

Key Insights and Ideas

  • Naturalness over Translation: The decision to collect questions directly from query logs rather than translating existing English datasets is crucial for capturing genuine linguistic nuances and avoiding unnatural phrasing.
  • Crowdsourcing Efficiency: Utilizing crowdsourcing for annotation significantly reduces costs while maintaining a reasonable level of quality, making dataset creation more scalable.
  • "NONE" Option Enhances Difficulty: The inclusion of an "unable to answer" option in JBoolQ makes the dataset more challenging and realistic, pushing the boundaries of current QA models.
  • Lexical Overlap as a Proxy for Annotation Artifacts: JNQ exhibits significantly lower lexical overlap between questions and paragraphs compared to JSQuAD, suggesting it is less susceptible to annotation artifacts and better reflects natural question-answering scenarios.
  • "How to" Questions are More Difficult: JNQ contains a higher proportion of "How to" questions compared to English NQ, indicating a greater challenge for models in understanding procedural or instructional queries.

Key Results

The paper reports baseline performance on the defined tasks:

  • Long Answer Extraction: Models trained on all data (including unlabeled negative examples) achieved higher precision than those trained only on hard negatives, highlighting the benefit of comprehensive training data. Human performance served as a strong benchmark.
  • Short Answer Extraction: Waseda-ROBERTa models showed promising results, though still significantly below human performance. The paper notes that some models struggled with extracting precise spans, sometimes returning entire paragraphs.
  • Yes/No Answer Identification: Models achieved high precision but relatively low recall, often defaulting to predicting "NONE." This indicates the challenge posed by the unbalanced data distribution and the "NONE" option. Human annotators demonstrated higher recall.
  • Open-Domain NQ: TF-IDF retrievers slightly outperformed DPR retrievers, possibly due to the relatively short questions and the presence of salient phrases that can be challenging for dense retrievers.

Limitations and Future Work

  • Human Performance as a Ceiling: The results consistently show a gap between model performance and human annotator performance, particularly in recall for yes/no identification and overall for short answer extraction.
  • Unsuitable Questions: The authors acknowledge that some questions in the dataset are inherently difficult for open-domain QA (e.g., subjective or opinion-based questions) and plan to exclude them in future work.
  • Further Exploration of Long Answer Extraction: The paper suggests exploring how to better utilize unlabeled paragraphs and investigate the performance of models on these potentially ambiguous cases.
  • Expanding Dataset Scope: The authors aim to facilitate further research on QA and NLP models in Japanese, implying that these datasets are a foundational step for more complex investigations.

Summary updated

BibTeX
@inproceedings{uematsu2024benchmark,
  author = {Takuya Uematsu and Hao Wang and Daisuke Kawahara and Tomohide Shibata},
  title = {{A Benchmark Suite of Japanese Natural Questions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure