Abstract
Relation Extraction (RE) is a task that identifies relationships between entities in a text, enabling the acquisition of relational facts and bridging the gap between natural language and structured knowledge. However, current RE models often rely on small datasets with low coverage of relation types, particularly when working with languages other than English. In this paper, we address these issues by providing two new resources for training and evaluating multilingual RE systems. First, we present SRED$^{\rm FM}$, an automatically annotated dataset covering 18 languages and 400 relation types, totaling more than 40 million triplet instances. Second, we propose RED$^{\rm FM}$, a smaller, human-revised dataset for seven languages. To demonstrate the utility of these datasets, we introduce mREBEL, the first end-to-end multilingual RE model that extracts triplets and entity types across multiple languages.
Introduction
Relation Extraction (RE) is fundamental to transforming unstructured text into structured knowledge, typically represented as triplets (e.g., [Paris, capital of, France]). These triplets are the building blocks of Knowledge Graphs. Despite its importance, the field of RE has long been hampered by a lack of high-quality, large-scale, and multilingual data.
Existing datasets are often restricted to English, cover a limited number of relation types, or rely on "noisy" distant supervision. Distant supervision assumes that if a knowledge base contains a fact, any sentence mentioning the involved entities must express that fact—an assumption that frequently fails in practice. Furthermore, most RE systems utilize a pipeline approach, where entity recognition and relation classification are separate steps, allowing errors to propagate from the first stage to the second. This work introduces a massive data resource and a unified model to address these challenges.
The SRED$^{\rm FM}$ and RED$^{\rm FM}$ Datasets
The authors provide two distinct resources to advance multilingual RE research:
SRED$^{\rm FM}$ (Silver-standard)
SRED$^{\rm FM}$ is a massive dataset generated through an expanded version of the cRocoDiLe pipeline, which aligns Wikipedia abstracts with Wikidata facts. It spans 18 languages and 400 relation types, encompassing over 40 million triplet instances. To ensure quality, the authors employed a "Triplet Critic" to filter out noise inherent in distant supervision. The dataset was also designed to address class imbalance, providing a more even distribution of relation types than previous benchmarks.
RED$^{\rm FM}$ (Gold-standard)
To provide a reliable benchmark, the authors created RED$^{\rm FM}$, a human-revised evaluation set. It covers 7 languages and 32 relation types. Through crowdsourcing, every triplet in this set was verified by humans, ensuring it represents a "gold standard" for evaluating the performance of multilingual RE models.
Compared to previous benchmarks like ACE05 or SMiLER, these datasets offer significantly greater scale and linguistic diversity, moving beyond the traditional focus on a handful of relations or Western languages.
Methodology
The creation of these resources and the subsequent model involved several innovative steps:
The Triplet Critic
To mitigate the noise of distant supervision, the researchers developed a "Triplet Critic" based on DeBERTaV3. This model functions similarly to a Natural Language Inference (NLI) system, evaluating whether a specific sentence actually entails the relationship suggested by the knowledge base. The authors found that approximately 8% of automatically generated triplets were "non-entailed." Removing this noise improved model performance by over 3 Micro-F1 points.
Multilingual Entity Typing
The researchers integrated entity typing into the extraction process. By leveraging BabelNet—a multilingual semantic network—and a Transformer-based classifier, they labeled entities with one of 13 types (e.g., Person, Organization, Celestial Body). This "silver-to-gold" refinement process ensured high accuracy in entity categorization across different languages.
The mREBEL Model
The authors introduced mREBEL, an end-to-end system that extends the REBEL architecture. Built on mBART-50, mREBEL uses a sequence-to-sequence (seq2seq) approach to "translate" raw sentences directly into a linearized string of triplets, including entity types. This unified approach prevents the error propagation common in pipeline systems.
Key Results and Insights
The utility of the new datasets and the mREBEL model was demonstrated through extensive experimentation:
- State-of-the-Art Performance: mREBEL outperformed existing multilingual baselines, such as HERBERTa, on the SMiLER benchmark by approximately 15 Micro-F1 points.
- End-to-End Efficiency: On the RED$^{\rm FM}$ test set, mREBEL achieved a Micro-F1 of 52.0. Notably, the model performed strongly even without fine-tuning on the human-annotated gold data, validating the high quality of the SRED$^{\rm FM}$ pre-training set.
- Zero-Shot Cross-Lingual Transfer: The Triplet Critic demonstrated a remarkable ability to generalize. Even when trained primarily on European languages, it performed well on unseen languages like Arabic and Chinese, suggesting that the logic of relation entailment is largely language-independent.
Limitations and Future Directions
While a significant step forward, the authors identify several areas for improvement. The datasets inherit the inherent biases of Wikipedia and Wikidata, such as better coverage for Western entities. Additionally, qualitative error analysis was limited in languages where the authors were not proficient, such as Arabic and Chinese.
Technical challenges also remain. Error analysis revealed that roughly 10% of mistakes were linked to coreference issues (e.g., the model failing to link a pronoun to its subject). Future work will focus on integrating coreference resolution and exploring alternative architectures, such as graph-based neural networks, to further enhance extraction performance.
Summary updated