ParsFEVER: a Dataset for Farsi Fact Extraction and Verification

Majid Zarharan, Mahsa Ghaderan, Amin Pourdabiri, Zahra Sayedi, Behrouz Minaei-Bidgoli, Sauleh Eetemadi, Mohammad Taher PilehvarACL 2021

TL;DR

This paper introduces ParsFEVER, the first publicly available dataset for Farsi fact extraction and verification, addressing the scarcity of resources for low-resource languages. By adapting the widely-used FEVER dataset's methodology, ParsFEVER offers a challenging benchmark with a significant proportion of "many-hop" claims, paving the way for improved fact-checking capabilities in Farsi.

What Problem Does This Paper Address?

The proliferation of misinformation poses significant societal challenges, making automated fact-checking a crucial area of research. However, developing robust fact-checking systems heavily relies on large, high-quality annotated datasets. Unfortunately, such datasets are predominantly available for high-resource languages like English, leaving many other languages, including Farsi, underserved. This gap hinders the development and evaluation of fact-checking technologies for a substantial portion of the global population.

Methods

The authors adapted the construction procedure of the English FEVER dataset to create ParsFEVER. Key methodological points include:

  • Claim Generation: Instead of random sentence selection as in FEVER, ParsFEVER annotators manually selected sentences from introductory sections of popular Farsi Wikipedia articles. This manual selection aimed to ensure claims were informative and to increase the likelihood of "many-hop" claims (requiring evidence from multiple sources). Claims were then mutated using six types of transformations (paraphrasing, negation, entity/relation substitution, generalization/specialization) to create a diverse set.
  • Claim Labeling: Annotators classified claims into SUPPORTED, REFUTED, or NOTENOUGHINFO based on provided evidence. They also identified the specific sentences used as evidence.
  • Annotation Tools: The authors developed their own Wikipedia crawler and annotation tools, as proprietary services used in FEVER were not publicly available.

Datasets

  • ParsFEVER: The primary contribution is this newly introduced dataset. It comprises approximately 23,000 manually annotated Farsi claims derived from 358 popular Farsi Wikipedia articles. A significant feature is that over 65% of its claims are "many-hop," a much higher proportion than the 13% found in FEVER.
  • FEVER: Used as a baseline and for comparison, FEVER is the standard English dataset for fact extraction and verification.

Key Insights and Ideas

  • Many-Hop Claims are Key: The deliberate focus on manual sentence selection and claim mutation in ParsFEVER resulted in a dataset with a substantially higher proportion of many-hop claims. This is a critical insight, as many-hop reasoning is a significant challenge for current fact-checking models, making ParsFEVER a more demanding benchmark than FEVER.
  • Low-Resource Doesn't Mean Low Quality: Despite a smaller training set compared to FEVER (about one-ninth), models trained on ParsFEVER achieve comparable downstream performance. This suggests that the quality of annotation and the nature of the claims (especially the many-hop aspect) are crucial factors for model effectiveness, not just sheer dataset size.
  • Adaptation is Necessary: The paper highlights that simply translating or directly applying methods from high-resource languages to low-resource languages is insufficient. The construction process needed specific adaptations to suit the Farsi language and cultural context, as well as to generate a more challenging dataset.

Key Results

  • Dataset Size: ParsFEVER contains approximately 23,000 manually annotated claims.
  • Many-Hop Claims: Over 65% of ParsFEVER claims are many-hop, compared to 13% in FEVER.
  • Model Performance: When evaluated on a held-out test set, a baseline fact-checking system (adapted from FEVER) achieved:
  • Claim Classification Accuracy: 50.0% on ParsFEVER (vs. 52.1% on FEVER).
  • Evidence Retrieval Accuracy: 28.1% on ParsFEVER (vs. 32.6% on FEVER). These results indicate that ParsFEVER is indeed a more challenging benchmark.

  • Inter-Annotator Agreement: The Fleiss' kappa score for ParsFEVER was 0.599, lower than FEVER's 0.684, which the authors attribute to the increased complexity of many-hop claims.

Limitations and Future Work

  • Limitations:
    • The inter-annotator agreement for ParsFEVER is lower than FEVER, suggesting the task is more challenging due to the higher proportion of many-hop claims.
  • The dataset is derived solely from Wikipedia, which might not fully represent the diversity of information sources in the real world.
  • Future Work:
    • The authors suggest using ParsFEVER as a benchmark for evaluating and analyzing existing fact-checking models, particularly those designed for many-hop reasoning.
  • This work can shed light on the ability of models to generalize beyond English and perform in low-resource settings.
  • Exploring other textual structures and corpora for fact extraction in Farsi is also a potential avenue.

Summary updated

BibTeX
@inproceedings{zarharan2021parsfever,
  author = {Majid Zarharan and Mahsa Ghaderan and Amin Pourdabiri and Zahra Sayedi and Behrouz Minaei-Bidgoli and Sauleh Eetemadi and Mohammad Taher Pilehvar},
  title = {{ParsFEVER: a Dataset for Farsi Fact Extraction and Verification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure