Neural Cross-Lingual Relation Extraction Based on Bilingual Word Embedding Mapping

Jian Ni, Radu FlorianEMNLP 2019 · DOI

TL;DR

This paper introduces a novel approach to cross-lingual relation extraction (RE) by mapping word embeddings from a target language to a source language (English). This allows a pre-trained English RE model to be directly applied to new languages, significantly reducing the need for language-specific resources and manual annotations. The method achieves strong performance across multiple languages with a small bilingual dictionary.

What Problem Does This Paper Address?

Relation extraction (RE) is a crucial task in Natural Language Processing (NLP) for identifying semantic relationships between entities in text. While neural network models have achieved state-of-the-art performance, they typically require large amounts of manually annotated data and language-specific tools (like dependency parsers) for each language. This makes it extremely challenging and expensive to build RE systems for low-resource languages. Existing cross-lingual methods often rely on parallel corpora or machine translation systems, which are not always readily available. This paper aims to bridge this gap by enabling direct transfer of an RE model from a high-resource language (English) to low-resource languages without requiring any target-language annotations or complex linguistic resources.

Methods

The core of the proposed approach lies in bilingual word embedding mapping. The authors leverage the observation that word embeddings from different languages often share similar geometric structures.

  1. Monolingual Word Embeddings: They first train word embeddings for both the source (English) and target languages independently using a variant of the Continuous Bag-of-Words (CBOW) model.
  2. Bilingual Mapping: A linear mapping (matrix $M_{t \to s}$) is learned to project target-language word embeddings into the source-language embedding space. This mapping is trained using a small bilingual dictionary (as few as 1,000 word pairs). They explore different mapping techniques, including regular least squares, orthogonal transformations, semi-supervised, and unsupervised methods, ultimately finding the regular mapping to be most effective for their task.
  3. Neural RE Model (English): A deep neural network RE model is built for the source language (English). This model uses word embeddings and generic language-independent features. It consists of an embedding layer, a context layer (Bi-LSTM or CNN), a summarization layer that pools contextual information around the entities, and an output layer for classification.
  4. Cross-Lingual Transfer: For a sentence in a target language, its word embeddings are first projected into the English embedding space using the learned mapping $M_{t \to s}$. The pre-trained English RE model is then applied to these projected embeddings to classify the relationship between entities.

Datasets

The paper evaluates its approach on two types of datasets:

  • In-house Dataset: Manually annotated RE data for 6 languages (English, German, Spanish, Italian, Japanese, Portuguese). This dataset defines 56 entity types and 53 relation types.
  • ACE05 Multilingual Dataset: Manually annotated RE data for 3 languages (English, Arabic, Chinese). This dataset defines 7 entity types and 6 relation types.

For cross-lingual evaluation, the English portion of these datasets serves as the source, and the other languages are treated as targets.

Key Insights and Ideas

  • Effective Cross-Lingual Transfer with Minimal Resources: The most significant insight is that a well-trained English RE model can be effectively transferred to other languages using only a small bilingual dictionary (1K word pairs) for embedding mapping, without needing any target-language annotated RE data or language-specific tools.
  • Bilingual Dictionary Size Matters, Up to a Point: The study demonstrates that increasing the size of the bilingual dictionary beyond 1,000 word pairs yields diminishing returns for cross-lingual RE performance, suggesting that a relatively small dictionary is sufficient for learning an effective mapping.
  • Regular Mapping Outperforms Orthogonal Mappings for RE: While orthogonal mappings have shown promise in word translation, the regular linear mapping proved more effective for cross-lingual RE. This might be because RE relies on semantic similarity captured by the embeddings, and orthogonal transformations can sometimes distort these relationships.
  • Language Family and Word Order Influence Transfer: The performance of cross-lingual transfer is correlated with the similarity in word order between the target language and the source language (English, SVO). Languages with similar word orders (e.g., Spanish, Italian) show higher relative accuracy compared to languages with different orders (e.g., Japanese, Arabic).

Key Results

  • The proposed cross-lingual RE approach achieves very good performance, reaching up to 79% of the accuracy of a supervised target-language RE model.
  • The Bi-LSTM context layer model generally outperforms the CNN context layer model for cross-lingual RE across most target languages.
  • For German, Spanish, Portuguese, and Chinese, the Bi-LSTM model achieves over 40.0 F1 scores.
  • The approach achieves over 75% relative accuracy for languages like German, Spanish, Italian, and Portuguese.
  • Even for more challenging languages like Japanese and Arabic, the method still achieves 55% and 52% of the supervised accuracy, respectively.
  • An ensemble of models further improves performance by 0.6-1.9 F1 points.

Limitations and Future Work

  • Language Family Dependence: The paper notes that performance is influenced by the word order similarity between the source and target languages. Languages with significantly different word orders (like Japanese and Arabic) are more challenging for transfer.
  • Future Work: The authors suggest that their approach provides a strong baseline for building cross-lingual RE models with minimal resources. Future work could explore more advanced mapping techniques or investigate how to better handle languages with very different linguistic structures. The paper also implicitly suggests that further research into unsupervised or semi-supervised mapping techniques could further reduce reliance on even small dictionaries.

Summary updated

BibTeX
@inproceedings{ni2019neural,
  author = {Jian Ni and Radu Florian},
  title = {{Neural Cross-Lingual Relation Extraction Based on Bilingual Word Embedding Mapping}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1038},
}
Figure