TL;DR
This paper introduces MILQA, the first publicly available extractive question answering benchmark for Hungarian, extending SQuAD 2.0 with more complex question types like list-based and yes/no questions. They also evaluate baseline retrieval and reader models, showing that while BM25 excels at retrieval, cross-lingual transfer from English significantly boosts reader model performance.
What Problem Does This Paper Address?
Developing robust question answering (QA) systems for low-resource languages like Hungarian is challenging due to a lack of high-quality, diverse datasets. Existing QA benchmarks, while foundational, often focus on simple extractive answers (a direct text span) and may not capture the nuances of real-world questions. This limits the ability to train models that can handle more complex queries, such as those requiring synthesis of information, list-based answers, or yes/no responses, especially in morphologically rich languages where direct text matching is less effective.
Methods
The authors created the MILQA dataset by largely following the principles of SQuAD 2.0 but introducing several innovations. These include:
- Diverse Question Types: Incorporating yes/no questions, list-like answers (potentially spanning multiple text segments), long answers (providing more context than just the minimal answer), and questions requiring simple calculations or reasoning.
- Pragmatic Answer Forms: For non-extractive questions, they provided pragmatically adequate answer forms to facilitate training generative models.
- Annotation Interface: They customized the Label Studio platform to support these new question types and answer formats, ensuring an intuitive workflow for annotators.
- Baseline Models: They implemented and evaluated both document retrieval models (BM25, vector-based) and reader models (answer span extraction). For reader models, they experimented with monolingual Hungarian models and multilingual models, testing cross-lingual transfer from English.
Datasets
- MILQA: The primary contribution is the creation of this new benchmark dataset for Hungarian question answering. It is based on high-quality Wikipedia articles and includes over 23,700 questions (17,000 answerable, 6,700 unanswerable). MILQA aims to be more challenging than SQuAD by including:
- Yes/no questions (9.20%)
- Non-SQuAD-compatible questions (over 30%) where answers are not single contiguous spans or require reformulation.
- Questions requiring simple arithmetic operations.
- Tricky unanswerable questions.
- SQuAD 2.0: MILQA's principles are largely derived from SQuAD 2.0, and cross-lingual transfer experiments utilize SQuAD 2.0.
- Other Benchmarks Mentioned: The paper also discusses various English QA datasets like Natural Questions (NQ), MS MARCO, and DROP, as well as multilingual datasets like MLQA and XQuAD, to contextualize their work.
Key Insights and Ideas
- Beyond Simple Spans: The paper strongly advocates for moving beyond the "single contiguous answer span" assumption of SQuAD, as it significantly limits the scope of questions that can be addressed, especially in morphologically rich languages.
- Cross-Lingual Transfer is Powerful: For reader models, leveraging pre-trained multilingual models and fine-tuning them on Hungarian data (or even English SQuAD data) proved to be a highly effective strategy, significantly outperforming models trained from scratch on limited Hungarian data.
- BM25 Still Reigns for Retrieval: Despite the rise of dense vector retrieval methods, the traditional BM25 algorithm, especially with lemmatization and PoS-based filtering, remained the superior choice for document retrieval in their experiments.
- Hungarian Morphology Matters: The paper highlights how Hungarian's rich morphology poses challenges for extractive QA, as answers might not appear in the exact form as in the text, necessitating more sophisticated handling than simple text matching.
Key Results
- Retrieval: BM25 with lemmatization and PoS filtering achieved the best retrieval performance (e.g., R@10 of 0.916, MRR@300 of 0.748). Vector-based models, including multilingual ones, generally underperformed BM25.
- Reader Models:
- Zero-shot performance of English-trained SQuAD models on MILQA was decent, with
xlmR-large-squad2achieving an F1 of 0.646 for short answers.
- Zero-shot performance of English-trained SQuAD models on MILQA was decent, with
- Fine-tuning Hungarian models (like
huBert-squadv2) significantly improved performance, reaching F1 scores of up to 0.768 for short answers. - Cross-lingual transfer from SQuAD 2.0 was crucial, with models fine-tuned on SQuAD 2.0 (e.g.,
huBert-squadv2-T) outperforming models trained from scratch. - Retro-Reader models showed promising results, outperforming others on the short answer task, though they are more computationally intensive.
Limitations and Future Work
- Dataset Size: While MILQA is a significant contribution, it is still limited in size compared to large English benchmarks.
- Model Complexity: The current work focuses on extractive QA. The authors acknowledge that generative models are needed for questions that cannot be answered extractively.
- Specific Question Types: The baseline experiments did not fully tackle multispan answers, questions requiring counting or arithmetic reasoning, or generative question answering.
- Future Work: The authors suggest further research into generative models, more advanced handling of multispan answers, and expanding the dataset's coverage. They also note the challenge of large language models "hallucinating" information, which extractive methods inherently avoid.
Summary updated