Abstract: Pere-Lluís Huguet Cabot, Simone Tedeschi, Axel-Cyrille Ngonga Ngomo, Roberto Navigli. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
2023.
Introduction
Presented at ACL 2023, this research introduces REDFM, a comprehensive suite of high-quality datasets and a novel end-to-end model designed to advance multilingual Information Extraction (IE). The work bridges the gap between raw natural language and structured knowledge across 18 languages by addressing the scarcity of high-quality multilingual data for Relation Extraction (RE).
The Data Bottleneck in Relation Extraction
Relation Extraction involves identifying "triplets" (subject, relation, object) within text—for example, extracting (Marie Curie, born in, Warsaw) from a biography. Despite its importance for building Knowledge Graphs, progress has been hindered by several "data bottlenecks":
- Language Bias: Most high-quality datasets are English-only.
- Low Coverage: Existing datasets often only cover a handful of relation types.
- Noisy Labels: Many large datasets use "distant supervision," which often results in false positives where a sentence mentions two entities but doesn't actually describe the relationship between them.
Methodology: The REDFM Pipeline
The researchers developed a multi-stage pipeline to generate cleaner, more diverse data and a robust extraction model.
Data Extraction and Filtering
- cRocoDiLe: An expanded pipeline that pulls millions of potential triplets from Wikipedia abstracts across 18 languages, linking them to Wikidata.
- The Triplet Critic: To solve the "noisy label" problem, the authors trained a cross-encoder model (based on DeBERTaV3) to act as a judge. This "Critic" evaluates whether a sentence actually supports a proposed relationship using textual entailment logic.
- Automated Entity Typing: The data is enriched with 13 entity types (e.g., Person, Location, Organization) using BabelNet and WordNet, a feature often missing in RE datasets.
The mREBEL Model
The researchers introduced mREBEL, the first end-to-end multilingual model capable of simultaneously extracting entity types and relationships. Based on mBART-50, mREBEL "reads" a sentence and "writes" out the triplets and entity types in a single pass, avoiding the error propagation common in multi-step pipelines.
Datasets: SRED<sup>FM</sup> and RED<sup>FM</sup>
The project released two primary resources:
- SRED<sup>FM</sup> (Silver-standard): A massive dataset covering 18 languages, 400 relation types, and over 44 million triplet instances. It is filtered by the Triplet Critic to ensure higher quality than standard distantly supervised sets.
- RED<sup>FM</sup> (Revised): A smaller, "gold-standard" dataset for 7 languages (Arabic, German, English, Spanish, French, Italian, and Chinese). This set was manually revised by human annotators to provide a reliable benchmark for evaluation.
Key Results and Insights
The Triplet Critic proved highly effective, achieving an average precision of 92.6% across languages. Its ability to perform "zero-shot" transfer to unseen languages like Arabic and Chinese suggests that the logic of textual support translates well across linguistic boundaries.
mREBEL demonstrated significant improvements over existing state-of-the-art models. On the SMiLER benchmark, it improved Micro-F1 scores by an average of 15 points. On the new RED<sup>FM</sup> test set, it established a strong baseline with a Micro-F1 of 52.0.
Conclusion
REDFM addresses critical gaps in multilingual IE. While successful, the authors note limitations such as grounding in existing Knowledge Graphs and challenges with coreference resolution, which accounted for 72.2% of errors in qualitative analysis. Future work will focus on integrating better coreference resolution and expanding the pipeline to more low-resource languages.
Summary updated