Benchmarks for Pirá 2.0, a Reading Comprehension Dataset about the Ocean, the Brazilian Coast, and Climate Change

Paulo Pirozelli, Marcos M. José, Igor Cataneo Silveira, Flávio Nakasato Cação, Sarajane M. Peres, Anarosa A. F. Brandão, Anna H. R. Costa, Fábio G. Cozman

Abstract

This paper introduces Pirá 2.0, a significantly improved and expanded bilingual (Portuguese-English) dataset designed to evaluate machine learning models on specialized scientific knowledge regarding oceanography and climate change. By establishing six distinct benchmarks—ranging from information retrieval to multiple-choice answering—the authors demonstrate that even state-of-the-art Large Language Models (LLMs) struggle with the technical complexity of this domain compared to generalized datasets. The results highlight an "expertise gap" and underscore the necessity of domain-specific context for accurate scientific reasoning.

Introduction

While Large Language Models (LLMs) like GPT-4 have demonstrated remarkable capabilities across general domains, their performance often falters when applied to "expert" or niche scientific fields. Most popular reading comprehension benchmarks, such as SQuAD, are built upon general knowledge from Wikipedia. To effectively evaluate a model's utility in scientific research, datasets must be domain-specific, bilingual to support global research communities, and rigorously curated to ensure data quality.

The authors address these needs by presenting Pirá 2.0, a dataset focused on the complex topics of oceanography and the Brazilian coast. This version improves upon the original Pirá 1.0 by addressing grammatical issues, removing redundancies, and introducing new features like answerability labels and automatic paraphrases.

Methodology and Benchmarks

The researchers transformed the original Pirá 1.0 into a robust evaluation framework by defining six distinct benchmarks to test different facets of model comprehension:

  1. Closed Generative QA (CGQA): Answering questions without provided context.
  2. Machine Reading Comprehension (MRC): Extracting or generating answers from a specific text.
  3. Information Retrieval (IR): Identifying correct supporting documents within a large corpus.
  4. Open Question Answering (OQA): A two-stage process of retrieving text and then answering.
  5. Answer Triggering (AT): Determining if a question is answerable based on the provided text.
  6. Multiple Choice QA (MCQA): Selecting the correct answer from five candidates.
Overview of the Pirá dataset evolution and the six evaluation benchmarks.

The Pirá 2.0 Dataset

Pirá 2.0 consists of 2,258 QA sets derived from scientific abstracts and United Nations reports. Unlike its predecessor, Pirá 2.0 underwent a rigorous curation process. Each entry includes supporting text, bilingual (English and Portuguese) questions and answers, and human-validated labels regarding the difficulty and sense of the questions.

Example of a Pirá 2.0 data entry showing supporting text, bilingual textual elements, and human evaluations.

To provide a baseline for difficulty, the authors compared Pirá 2.0 against standard datasets like SQuAD 1.1, SQuAD 2.0, and Natural Questions. The comparison revealed that Pirá 2.0 is significantly more challenging for both humans and machines due to its technical nature.

Key Results and Discussion

The Expertise Gap

Human F1-scores on Pirá 2.0 hover between 51 and 54, which is notably lower than the 86–89 range typically seen on SQuAD. This suggests that the technical complexity of oceanographic text makes it inherently difficult for even human experts to reach a consensus on specific answer spans, setting a high bar for automated systems.

The Necessity of Context

In the CGQA benchmark, where models must answer without external context, performance was poor (F1 < 17). This indicates that LLMs have not "memorized" this specific scientific niche during pre-training and require external documentation (e.g., via Retrieval-Augmented Generation) to provide accurate information.

Retrieval and Model Performance

In the Information Retrieval task, traditional keyword-based methods like BM25 consistently outperformed dense retrieval methods like DPR. This is likely because scientific discourse relies on highly specific terminology that keyword matching captures more effectively than general semantic embeddings.

Furthermore, while GPT-4 performed well, smaller specialized models like UnifiedQA matched or surpassed it in the Multiple Choice benchmark when fine-tuned specifically on Pirá 2.0. In the MRC task, T5-base achieved the highest F1-score for English (51.27), while PTT5-base led for Portuguese (27.90), both remaining well below human baselines.

Conclusion

Pirá 2.0 serves as a challenging new frontier for reading comprehension models. While the dataset's size (2,258 samples) is modest, its high-quality curation and bilingual nature make it a valuable tool for testing domain-specific expertise. Future work will focus on data augmentation and the development of hybrid pipelines that combine the search capabilities of traditional retrieval with the linguistic polish of modern LLMs.

Summary updated

BibTeX
@article{pirozelli2024benchmarks,
  author = {Paulo Pirozelli and Marcos M. José and Igor Cataneo Silveira and Flávio Nakasato Cação and Sarajane M. Peres and Anarosa A. F. Brandão and Anna H. R. Costa and Fábio G. Cozman},
  title = {{Benchmarks for Pirá 2.0, a Reading Comprehension Dataset about the Ocean, the Brazilian Coast, and Climate Change}},
  journal = {Data Intell.},
  year = {2024},
  doi = {10.1162/dint\_a\_00245},
}
Figure