Automated Fact-Checking in Brazilian Portuguese: Resources and Baselines

Marcelo Mussi Delucis, Lucas Fraga, Otávio Parraga, Christian Mattjie de Oliveira, Rafaela Cappelari Ravazio, Rodrigo C. Barros, Lucas S. Kupssinskü

Abstract

Automated fact-checking is essential for mitigating misinformation, yet resources for non-English languages remain scarce. This paper presents LIAR-BR and AVERITEC-BR, the first sophisticated, multi-class fact-checking datasets for Brazilian Portuguese (PTBR), translated from prominent English benchmarks. We evaluate various modeling strategies, including zero-shot and few-shot prompting with Gemma 3 and fine-tuning encoder models like mBERT and BERTimbau. Our findings indicate that while large language models (LLMs) struggle with veracity classification in a zero-shot setting, fine-tuned models—especially those pre-trained on native PTBR text—achieve superior performance when provided with claim metadata or gold evidence.

Introduction

The development of automated fact-checking tools is heavily skewed toward English, leaving "low-resource" languages like Brazilian Portuguese at a disadvantage. Existing PTBR datasets often oversimplify fact-checking as a binary "True vs. False" task. However, real-world misinformation frequently involves nuances such as half-truths and context-dependent claims. This research addresses the gap by introducing high-quality, multi-class datasets that incorporate evidence-based reasoning, providing a more robust foundation for fact-checking in the Portuguese-speaking world.

Methodology

The researchers employed a two-stage approach: resource creation via translation and baseline model establishment.

Resource Creation

The English source datasets were translated into PTBR using Gemma 3 (4B parameters). To maintain high quality, a zero-shot prompting strategy was used to translate individual fields (claims, justifications, speaker information) while preserving technical metadata like URLs and timestamps.

Modeling Strategies

Three primary approaches were evaluated:

  1. Zero-shot prompting: Utilizing Gemma 3 to classify claims without prior examples.
  2. Few-shot prompting: Providing Gemma 3 with labeled examples for each class.
  3. Fine-tuning: Training encoder-based models—mBERT, BERT-Large, and BERTimbau-Large (a Portuguese-specific BERT)—directly on the translated datasets.

Datasets

The study introduces PTBR versions of two landmark datasets:

  • LIAR-BR: Comprising 12,836 short political statements, this dataset uses a six-point veracity scale (from "Pants on Fire" to "True") and includes rich speaker metadata.
  • AVERITEC-BR: Consisting of 4,568 real-world claims, this dataset pairs claims with "Gold Evidence" in a question-answer (QA) format, requiring models to reason based on provided facts.

Results and Analysis

The experimental results highlight several critical insights regarding model performance and the necessity of contextual information.

The Impact of Evidence and Metadata

Models trained solely on claim text reached a performance ceiling of approximately 25% accuracy. Significant improvements were observed when models were granted access to external context. In the LIAR-BR task, the inclusion of speaker metadata nearly doubled performance metrics. Similarly, in AVERITEC-BR, the "Gold Evidence" scenario allowed models to significantly outperform text-only baselines.

Performance on AVERITEC and AVERITEC-BR datasets across different evidence scenarios and models.

Fine-Tuning vs. Large-Scale Prompting

A key finding was that the 4-billion-parameter Gemma 3 model, even with few-shot prompting, was outperformed by smaller, fine-tuned BERT models. This suggests that task-specific gradient-based learning remains more effective for specialized veracity classification than general-purpose LLM reasoning.

Native Pre-training Advantage

BERTimbau-Large, pre-trained specifically on Portuguese corpora, consistently outperformed both the multilingual mBERT and the English-centric BERT-Large. This underscores the value of language-specific pre-training for capturing local linguistic nuances.

Performance on LIAR and LIAR-BR datasets comparing statement-only and metadata-enhanced approaches.

Error Patterns

Error analysis revealed that most models exhibited a "Mode Error" of 1. This indicates that when models misclassified a claim, they typically chose a label adjacent to the correct one on the ordinal scale (e.g., "Mostly-True" instead of "Half-True"). This suggests the models are capturing the underlying spectrum of truthfulness, even when failing to identify the exact category.

Conclusion and Future Work

While these translated datasets provide a vital starting point, they are limited by their U.S.-centric content, reflecting North American political discourse. Future research will focus on:

  1. Developing native PTBR datasets that reflect local Brazilian misinformation patterns and regional issues.
  2. Integrating automated retrieval systems to enable models to find evidence dynamically.
  3. Evaluating larger multilingual LLMs to determine if increased scale can eventually bridge the gap with fine-tuned encoder models.

Summary updated

Abstract

Resumo The spread of misinformation presents a growing societal challenge, particularly in low-resource languages such as Brazilian Portuguese (PTBR), where the scarcity of high-quality datasets limits automated fact-checking tools. In this work, we introduce translated PTBR versions of two influential English-language fact-checking datasets: LIAR and AVERITEC. These resources support multi-class veracity classification and incorporate evidence-based reasoning. We also establish baseline results for both datasets using a range of model configurations, including zero-shot and few-shot prompting with Gemma 3, and fine-tuning of encoder-based models such as mBERT, BERT-Large, and BERTimbau-Large. Across both datasets, fine-tuned encoder-based models consistently outperformed Gemma 3 in zero-shot and few-shot settings. Our results underscore the importance of task-specific fine-tuning and evidence inclusion for veracity classification in PTBR. All datasets, translation scripts, and evaluation protocols are publicly released to support further research in this area.
BibTeX
@inproceedings{delucis2025automated,
  author = {Marcelo Mussi Delucis and Lucas Fraga and Otávio Parraga and Christian Mattjie de Oliveira and Rafaela Cappelari Ravazio and Rodrigo C. Barros and Lucas S. Kupssinskü},
  title = {{Automated Fact-Checking in Brazilian Portuguese: Resources and Baselines}},
  booktitle = {Stil},
  year = {2025},
  doi = {10.5753/stil.2025.37820},
}
Figure