Multilingual Open Relation Extraction Using Cross-lingual Projection

Manaal Faruqui, Shankar Kumar

TL;DR

Researchers have developed a pipeline to perform Open Relation Extraction (RE) in 61 different languages without needing language-specific tools like dependency parsers. By translating sentences to English, extracting relations using established English tools, and "projecting" those relations back to the original language using a novel BLEU-based alignment algorithm, they’ve made structured knowledge extraction possible for low-resource languages.

What Problem Does This Paper Address?

In the world of Information Extraction, Open Relation Extraction (RE) is the "wild west." Unlike closed RE, which looks for a fixed set of relationships (e.g., BornIn or WorksFor), Open RE identifies any arbitrary phrase that connects two arguments (e.g., "is the lead guitarist of" or "was conscripted to").

The challenge? Most state-of-the-art Open RE systems are "English-only" because they rely heavily on high-quality linguistic "plumbing"—specifically Part-of-Speech (POS) taggers and dependency parsers. For the vast majority of the world's 7,000+ languages, these tools either don't exist or aren't accurate enough to be useful. This creates a massive bottleneck for building global knowledge bases from sources like Wikipedia.

Methods

The authors propose a three-step "Cross-lingual Projection" pipeline that bypasses the need for native linguistic tools:

  1. Translation: Translate a source sentence (e.g., in Spanish) into English using a Machine Translation (MT) system (like Google Translate), which also provides word-to-word alignments.
  2. English Extraction: Run a robust English Open RE system—in this case, OLLIE—on the translated sentence to extract triples in the format (arg1; relation; arg2).
  3. Projection (The Secret Sauce): Map the English arguments and relation phrases back to the original source sentence.

The standard way to do this is "naive word alignment," but that often results in "broken," non-contiguous phrases that make no sense in the source language. To fix this, the authors introduced a novel projection algorithm that uses BLEU scores (a metric usually used to evaluate translation quality) to find the most similar contiguous phrase in the source language that overlaps with the aligned words. This ensures the extracted relations are grammatically and semantically coherent in their native tongue.

Datasets

  • Wikipedia Corpus: The authors applied their pipeline to the entire Wikipedia corpus for 61 languages, extracting millions of relations.
  • Manual Evaluation Sets: To verify their work, they hired professional linguists to manually annotate 1,000 randomly selected relations for three typologically diverse languages: French, Hindi, and Russian.
  • Public Release: The manual annotations and the massive 61-language dataset were released publicly (originally hosted on Kaggle and CMU's servers).

Key Insights and Ideas

  • Leveraging MT over Parsers: The core "aha moment" is that it is currently easier to build a decent Machine Translation system for a new language than it is to build a high-quality dependency parser. By piggybacking on MT, RE becomes "language independent."
  • Contiguity is King: Naive projection often produces "word salad." By forcing the system to select contiguous phrases via their BLEU-based algorithm, the researchers significantly improved the readability and utility of the extracted data.
  • The Length-Accuracy Trade-off: The researchers found that the length of the relation phrase is inversely correlated with the BLEU score. Russian relations were short and easy to project accurately, while Hindi and French relations were longer and more complex, leading to lower similarity scores compared to human gold standards.

Key Results

The system's performance was evaluated based on the percentage of "valid" extractions (as judged by humans) and the BLEU score similarity to human-selected phrases:

  • French: Achieved the highest validity at 81.6%, with a BLEU score of 0.47.
  • Hindi: Had a validity of 64.9% and a BLEU score of 0.38.
  • Russian: Had a validity of 63.5%, but the highest BLEU score (0.62), likely because Russian relation phrases were significantly shorter (averaging 1.8 words) than French (3.6) or Hindi (4.1).
  • Scale: The researchers successfully scaled the system to 61 languages, producing a massive multilingual knowledge resource. For example, they extracted over 7.5 million relations for Russian and 6.7 million for French.

Limitations and Future Work

  • MT Dependency: The system is only as good as the Machine Translation. If the MT system mangles the translation, the extraction will be garbage.
  • Binary Judgments: The evaluation used a simple valid/invalid (1/0) label, which the authors admit is coarse and might unfairly penalize relations that are semantically correct but slightly ungrammatical.
  • Ignoring Context: The system currently ignores "clausal information" (e.g., "Early astronomers believed that...").
  • Future Directions: The authors suggest using these automatically projected relations as "silver standard" training data to train native Open RE systems in other languages, potentially removing the need for the translation step in the long run.

Summary updated

Abstract

Open domain relation extraction systems identify relation and argument phrases in a sentence without relying on any underlying schema. However, current state-of-the-art relation extraction systems are available only for English because of their heavy reliance on linguistic tools such as part-of-speech taggers and dependency parsers. We present a cross-lingual annotation projection method for language independent relation extraction. We evaluate our method on a manually annotated test set and present results on three typologically different languages. We release these manual annotations and extracted relations in 61 languages from Wikipedia.
BibTeX
@inproceedings{faruqui2015multilingual,
  author = {Manaal Faruqui and Shankar Kumar},
  title = {{Multilingual Open Relation Extraction Using Cross-lingual Projection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1151},
}

Loading related content...

Figure