Multilingual Open Relation Extraction Using Cross-lingual Projection

Manaal Faruqui, Shankar Kumar

Abstract

Open domain relation extraction systems identify relation and argument phrases in a sentence without relying on any underlying schema. However, current state-of-the-art relation extraction systems are available only for English because of their heavy reliance on linguistic tools such as part-of-speech taggers and dependency parsers. We present a cross-lingual annotation projection method for language independent relation extraction. We evaluate our method on a manually annotated test set and present results on three typologically different languages. We release these manual annotations and extracted relations in 61 languages from Wikipedia.

Introduction

Open Relation Extraction (RE) is a critical task in information extraction that aims to identify arbitrary phrases representing relationships between arguments in text. Unlike closed RE, which operates on a fixed schema, Open RE is more flexible but significantly more dependent on high-quality linguistic preprocessing, such as dependency parsing and part-of-speech tagging. Consequently, most advanced Open RE systems are restricted to English, leaving a vast majority of the world's languages underserved. This research addresses this bottleneck by introducing a cross-lingual projection method that leverages existing English-centric tools for multilingual extraction.

Methodology: Cross-lingual Projection

The proposed pipeline bypasses the need for native linguistic tools in the target language by utilizing Machine Translation (MT) as a bridge. The process consists of three primary stages:

  1. Translation and Alignment: A source sentence is translated into English using a standard MT system. This step also generates word-to-word alignments between the source and translated text.
  2. English Extraction: An established English Open RE system, OLLIE, is applied to the translated sentence to extract triples in the form (arg1; relation; arg2).
  3. Projection: The extracted English phrases are mapped back to the source language. To avoid the "word salad" effect of naive alignment, a novel algorithm selects the most similar contiguous phrase in the source language based on BLEU scores.
The three-step pipeline: Translation, English Extraction, and Projection.

Experimental Setup and Data

The system was applied to the entire Wikipedia corpus for 61 languages, creating a massive multilingual knowledge resource. For evaluation, professional linguists manually annotated 1,000 relations for three typologically diverse languages: French, Hindi, and Russian. These manual annotations serve as a gold standard to measure the validity and accuracy of the projected relations.

Results and Discussion

The performance of the projection method was evaluated using two metrics: the percentage of extractions judged "valid" by humans and the BLEU score similarity between automatically projected phrases and human-selected "gold" phrases.

Performance metrics across French, Hindi, and Russian, including validity and BLEU scores.

As shown in the results, French achieved the highest validity at 81.6%. Interestingly, while Russian had a lower validity rate (63.5%), it achieved the highest BLEU score (0.62). This discrepancy is attributed to the length of the relation phrases; Russian relations were significantly shorter (averaging 1.8 words) compared to French (3.6) or Hindi (4.1), making them easier to project accurately. The study highlights a clear trade-off: longer, more complex relations in languages like Hindi are harder to project coherently, leading to lower similarity scores.

Conclusion

This work demonstrates that cross-lingual projection is a viable strategy for scaling Open RE to low-resource languages without requiring native dependency parsers. By leveraging the maturity of MT systems, the researchers successfully extracted millions of relations across 61 languages. Future work involves using these projected "silver standard" relations to train native Open RE models, potentially eliminating the translation step in the future.

Summary updated

Abstract

Open domain relation extraction systems identify relation and argument phrases in a sentence without relying on any underlying schema. However, current state-of-the-art relation extraction systems are available only for English because of their heavy reliance on linguistic tools such as part-of-speech taggers and dependency parsers. We present a cross-lingual annotation projection method for language independent relation extraction. We evaluate our method on a manually annotated test set and present results on three typologically different languages. We release these manual annotations and extracted relations in 61 languages from Wikipedia.
BibTeX
@inproceedings{faruqui2015multilingual,
  author = {Manaal Faruqui and Shankar Kumar},
  title = {{Multilingual Open Relation Extraction Using Cross-lingual Projection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1151},
}
Figure