Introduction
Portuguese is the sixth most spoken language globally, yet it remains significantly underserved in the field of Natural Language Processing (NLP), often categorized as a "resource-poor" language. A primary bottleneck is the scarcity of extractive Question-Answering (QA) datasets, where answers are direct spans from a source text. Prior to this research, the only substantial extractive dataset for Portuguese was FaQuAD, which contains fewer than 1,000 pairs.
This paper addresses the challenge of Question Answer Generation (QAG)—the task of identifying viable answer spans within a text and generating corresponding questions. QAG is a foundational component for developing automated educational tools, generating synthetic training data for search engines, and creating practice assessments.
Methodology
The study evaluates two primary architectural frameworks using PTT5, a T5 model pre-trained specifically on Portuguese corpora:
- Pipeline (Pipe): A decoupled approach where the model first identifies a candidate answer span and subsequently generates a question for that span.
- End-to-End (E2E): A unified approach where the model processes the entire text to output multiple question-answer pairs in a single pass.
These specialized models were benchmarked against state-of-the-art Large Language Models (LLMs), including GPT-3.5-turbo and Sabiá-medium, using few-shot prompting. The researchers also investigated the impact of decoding strategies: Beam Search, which prioritizes high-probability sequences for accuracy, and Top-p Nucleus Sampling, which introduces randomness to enhance linguistic diversity. To ensure quality, various filtering mechanisms were employed, most notably "Roundtrip consistency," which validates whether a secondary model can correctly answer the generated question using the original answer span.
Data Resources
The researchers contributed three new human-validated extractive QA datasets, comprising 993 high-quality pairs:
- Wiki: 692 questions derived from Portuguese Wikipedia passages.
- News: 599 questions based on Brazilian news articles from early 2024.
- geoREGIS: 528 questions from technical geoscientific documents in the oil and gas sector.
Additionally, the team produced PT-SQuAD, a high-fidelity translation of the SQuAD 1.1 dataset, meticulously aligned to preserve the integrity of answer spans within the translated text.
Results and Analysis
The Accuracy-Diversity Trade-off
A central finding of the study is the "Accuracy-Diversity Paradox." While Beam Search consistently yields the highest accuracy scores, it tends to produce repetitive and formulaic questions. Conversely, Top-p sampling generates more varied and human-like queries but at the cost of a higher error rate. This trade-off is critical for practitioners: high-stakes applications may require the reliability of Beam Search, while educational tools might benefit from the variety provided by Top-p sampling.
As shown in the performance table, the Pipeline approach significantly outperformed the End-to-End models. For instance, on the Wiki dataset, the PTT5 Pipeline achieved an accuracy of 0.73, nearly doubling the 0.43 accuracy of the E2E variant. The E2E models frequently failed to maintain the logical connection between the selected answer and the generated question.
Domain Sensitivity and Answer Classification
The nature of the generated questions varied significantly across domains. In the News corpus, models focused heavily on "Names" (entities), whereas the technical geoREGIS corpus shifted the focus toward "Descriptions" of processes and formations.
LLM Performance and Error Profiles
GPT-3.5-turbo and Sabiá-medium demonstrated superior performance, with GPT-3.5 achieving accuracy rates between 80% and 93%. However, error analysis revealed that LLMs are prone to "hallucinations," often subtly altering the wording of an answer so it is no longer strictly extractive. The most frequent error across all models was "Off-target" generation, where the question was either unanswerable from the text or pointed to an incorrect span.
Conclusion
This research highlights the effectiveness of pipeline-based architectures and the current dominance of LLMs in Portuguese QAG. While filtering methods like Answer Scoring improve dataset quality, they often inadvertently remove complex questions, leading to a bias toward simpler queries. Future work will explore non-extractive QAG—where answers are synthesized rather than copied—and the development of more sophisticated linguistic filters to capture subtle grammatical errors.
Summary updated