Introduction
Natural Language Processing (NLP) has historically been dominated by English-centric benchmarks. The Stanford Question Answering Dataset (SQuAD) revolutionized reading comprehension tasks, but similar high-quality, large-scale resources for other languages have been lacking. For the French language, researchers previously relied on machine-translated datasets or cross-lingual transfer from English models. However, these methods often fail to capture linguistic nuances or reach the performance levels of models trained on native data. FQuAD (French Question Answering Dataset) addresses this gap by providing the first large-scale, high-quality native French Reading Comprehension dataset.
Dataset Construction
FQuAD follows the rigorous methodology established by SQuAD 1.1. The authors sampled high-quality French Wikipedia articles and employed French university students for human annotation. Annotators were tasked with writing questions and identifying the precise "span" of text within a paragraph that provides the answer.
The dataset was developed in two stages. While FQuAD 1.0 provided an initial baseline, FQuAD 1.1 significantly expanded the scope to over 60,000 question-answer pairs across 326 articles. This version specifically targeted increased complexity, encouraging questions that require deeper reasoning (e.g., "Why" or "How" questions) and answers involving diverse parts of speech like verbs and adjectives.
Key Insights
A primary finding of this work is the "Native Premium": models trained on native French data significantly outperform those trained on translated English data. Specifically, native training yields an improvement of approximately 10 points in Exact Match (EM) scores compared to translation-based approaches.
Linguistic analysis also revealed that French answers in FQuAD are notably longer than their English counterparts in SQuAD (averaging 4.24 tokens versus 2.72 tokens). This increased length makes the Exact Match metric more challenging, as models must precisely identify longer spans without missing articles or trailing words.
Experimental Results
The researchers benchmarked FQuAD using several state-of-the-art transformer models. The results demonstrate that monolingual models, such as CamemBERT, consistently outperform multilingual models like mBERT or XLM-RoBERTa when native training data is available. This suggests that specialized language modeling holds a distinct edge over universal approaches for high-stakes tasks.
Notably, the CamemBERT-Large model achieved a new state-of-the-art for French reading comprehension, achieving an F1 score of 92.2 and an EM of 82.1. This performance actually surpasses human performance on the Exact Match metric (75.9%), highlighting the model's consistency in identifying answer boundaries as defined by the annotation guidelines.
Discussion and Limitations
While FQuAD 1.1 represents a major milestone for French NLP, it has limitations. Like SQuAD 1.1, it currently lacks "unanswerable" questions, which can lead to model hallucinations where a system forces an answer even when none exists. Future iterations aim to include such samples to improve robustness. Additionally, the dataset is currently limited to the Wikipedia domain; performance on informal or highly specialized text (e.g., legal or medical) remains an area for future exploration. Despite these limitations, FQuAD provides a critical foundation for developing robust, production-grade French language models.
Summary updated