Abstract
Automated fact-checking is essential for mitigating misinformation, yet resources for non-English languages remain scarce. This paper presents LIAR-BR and AVERITEC-BR, the first sophisticated, multi-class fact-checking datasets for Brazilian Portuguese (PTBR), translated from prominent English benchmarks. We evaluate various modeling strategies, including zero-shot and few-shot prompting with Gemma 3 and fine-tuning encoder models like mBERT and BERTimbau. Our findings indicate that while large language models (LLMs) struggle with veracity classification in a zero-shot setting, fine-tuned models—especially those pre-trained on native PTBR text—achieve superior performance when provided with claim metadata or gold evidence.
Introduction
The development of automated fact-checking tools is heavily skewed toward English, leaving "low-resource" languages like Brazilian Portuguese at a disadvantage. Existing PTBR datasets often oversimplify fact-checking as a binary "True vs. False" task. However, real-world misinformation frequently involves nuances such as half-truths and context-dependent claims. This research addresses the gap by introducing high-quality, multi-class datasets that incorporate evidence-based reasoning, providing a more robust foundation for fact-checking in the Portuguese-speaking world.
Methodology
The researchers employed a two-stage approach: resource creation via translation and baseline model establishment.
Resource Creation
The English source datasets were translated into PTBR using Gemma 3 (4B parameters). To maintain high quality, a zero-shot prompting strategy was used to translate individual fields (claims, justifications, speaker information) while preserving technical metadata like URLs and timestamps.
Modeling Strategies
Three primary approaches were evaluated:
- Zero-shot prompting: Utilizing Gemma 3 to classify claims without prior examples.
- Few-shot prompting: Providing Gemma 3 with labeled examples for each class.
- Fine-tuning: Training encoder-based models—mBERT, BERT-Large, and BERTimbau-Large (a Portuguese-specific BERT)—directly on the translated datasets.
Datasets
The study introduces PTBR versions of two landmark datasets:
- LIAR-BR: Comprising 12,836 short political statements, this dataset uses a six-point veracity scale (from "Pants on Fire" to "True") and includes rich speaker metadata.
- AVERITEC-BR: Consisting of 4,568 real-world claims, this dataset pairs claims with "Gold Evidence" in a question-answer (QA) format, requiring models to reason based on provided facts.
Results and Analysis
The experimental results highlight several critical insights regarding model performance and the necessity of contextual information.
The Impact of Evidence and Metadata
Models trained solely on claim text reached a performance ceiling of approximately 25% accuracy. Significant improvements were observed when models were granted access to external context. In the LIAR-BR task, the inclusion of speaker metadata nearly doubled performance metrics. Similarly, in AVERITEC-BR, the "Gold Evidence" scenario allowed models to significantly outperform text-only baselines.
Fine-Tuning vs. Large-Scale Prompting
A key finding was that the 4-billion-parameter Gemma 3 model, even with few-shot prompting, was outperformed by smaller, fine-tuned BERT models. This suggests that task-specific gradient-based learning remains more effective for specialized veracity classification than general-purpose LLM reasoning.
Native Pre-training Advantage
BERTimbau-Large, pre-trained specifically on Portuguese corpora, consistently outperformed both the multilingual mBERT and the English-centric BERT-Large. This underscores the value of language-specific pre-training for capturing local linguistic nuances.
Error Patterns
Error analysis revealed that most models exhibited a "Mode Error" of 1. This indicates that when models misclassified a claim, they typically chose a label adjacent to the correct one on the ordinal scale (e.g., "Mostly-True" instead of "Half-True"). This suggests the models are capturing the underlying spectrum of truthfulness, even when failing to identify the exact category.
Conclusion and Future Work
While these translated datasets provide a vital starting point, they are limited by their U.S.-centric content, reflecting North American political discourse. Future research will focus on:
- Developing native PTBR datasets that reflect local Brazilian misinformation patterns and regional issues.
- Integrating automated retrieval systems to enable models to find evidence dynamically.
- Evaluating larger multilingual LLMs to determine if increased scale can eventually bridge the gap with fine-tuned encoder models.
Summary updated