Bilingual Word Embeddings from Parallel and Non-parallel Corpora for Cross-Language Text Classification

Aditya Mogadala, Achim RettingerNAACL 2016

TL;DR

This paper introduces BRAVE, a novel method for learning bilingual word embeddings that can be trained on either parallel or non-parallel corpora. BRAVE leverages paragraph vectors to capture sentence-level semantics and outperforms existing methods on cross-language text classification tasks, demonstrating its effectiveness in bridging language barriers for NLP.

What Problem Does This Paper Address?

Natural Language Processing (NLP) tasks often suffer from a lack of labeled data, especially for low-resource languages. While monolingual word embeddings have advanced significantly, they don't directly address cross-language challenges. Transferring knowledge from high-resource languages is a promising avenue, but existing methods for learning bilingual embeddings often rely on strictly aligned parallel corpora, which are scarce, or struggle to capture nuanced semantic information beyond simple word co-occurrences. This paper tackles the challenge of learning effective bilingual word embeddings that can generalize across languages for tasks like text classification, even when only non-parallel or weakly aligned data is available.

Methods

The core of BRAVE lies in learning bilingual word embeddings by jointly optimizing a monolingual objective with a bilingual regularization function.

  • Monolingual Objective (M(·)): BRAVE utilizes the paragraph vector distributed memory model (PV-DM) (Le and Mikolov, 2014) to learn fixed-length vector representations for sentences or documents. This captures compositional meaning beyond bag-of-words approaches.
  • Bilingual Regularization (Φ(·)): This is where BRAVE offers flexibility. It can be adapted based on the available data:
  • BRAVE-S (Sentence-Aligned Parallel Corpora): This variant uses sentence-aligned parallel corpora. It employs an elastic net regularization, combining L2-loss between sentence paragraph vectors and L2-loss between the mean of word vectors within sentence pairs. This approach captures both compositional and non-compositional meanings and avoids the need for explicit word alignment.
  • BRAVE-D (Non-Parallel Document Corpora): This variant leverages label/class-aligned non-parallel documents. It involves a two-step process:
  • Manifold Alignment: Procrustes analysis is used to find partial alignment between documents of the same class in different languages.
  • Bilingual Cues: Pairs of partially aligned documents are then used to extract bilingual cues, similar to BRAVE-S, but with paragraph vectors learned for entire documents. This allows for the use of readily available datasets like sentiment-labeled reviews.

Datasets

The paper uses several standard datasets for evaluation:

  • RCV1/RCV2 Corpora: A subset of this corpus is used for the Cross-Language Document Classification (CLDC) task.
  • TED Corpus: A subset of this corpus, containing transcriptions and translations across 12 languages, is used for multi-label CLDC.
  • Amazon Product Reviews (CL-APR): This dataset, containing sentiment-labeled reviews across English, German, and French, is used for Cross-Language Sentiment Classification (CLSC).
  • Europarl-v7 (EP): This corpus is used as both monolingual and parallel training data for BRAVE-S.

Key Insights and Ideas

  • Flexibility in Data Usage: The most significant insight is BRAVE's ability to learn effective bilingual embeddings from either sentence-aligned parallel corpora or label/class-aligned non-parallel corpora. This greatly expands the applicability of cross-lingual NLP to languages with limited parallel data.
  • Paragraph Vectors for Richer Semantics: By incorporating paragraph vectors, BRAVE moves beyond simple word co-occurrence statistics, capturing more nuanced sentence-level and document-level semantics, which is crucial for tasks like sentiment analysis.
  • Elastic Net for Robustness: The use of elastic net regularization in BRAVE-S helps to balance the influence of monolingual and cross-lingual objectives, leading to more robust embeddings.
  • Label-Aligned Corpora are Crucial for Specific Tasks: The performance of BRAVE-D highlights the importance of using label/class-aligned corpora that are relevant to the target task (e.g., sentiment-labeled data for sentiment classification).

Key Results

  • State-of-the-Art Performance: BRAVE-S and BRAVE-D consistently outperform several baseline methods on the evaluated Cross-Language Text Classification (CLTC) tasks, including CLDC, multi-label CLDC, and CLSC.
  • RCV1/RCV2 CLDC: BRAVE-S achieves competitive results, outperforming many existing embedding models.
  • TED Corpus Multi-label CLDC: BRAVE-S (trained on TED corpus) shows better linguistic regularities specific to the corpus compared to general-purpose bilingual embeddings.
  • Amazon Reviews CLSC: BRAVE-D demonstrates strong performance, proving more suitable for sentiment classification due to its ability to leverage sentiment-specific label information from non-parallel corpora. It also shows less parameter dependency compared to some prior methods.
  • Qualitative Analysis: Table 5 shows that BRAVE-D can identify better sentiment-related word neighbors in cross-lingual settings compared to BRAVE-S.

Limitations and Future Work

  • BRAVE-D Performance on Unrelated Domains: The paper notes that BRAVE-D's performance can be suboptimal when the label/class-aligned corpora have minimal overlap with the target domain (e.g., using sentiment data for a different classification task). This underscores the importance of domain relevance in label-aligned corpora.
  • Asymmetry and Language-Specific Words: In some cases, BRAVE models could not outperform machine translation baselines, potentially due to language-specific words that lack direct equivalents.
  • Future Work: The authors aim to extend the approach to learn multilingual semantic spaces with more labels/classes, suggesting a path towards more comprehensive cross-lingual understanding.

Summary updated

BibTeX
@inproceedings{mogadala2016bilingual,
  author = {Aditya Mogadala and Achim Rettinger},
  title = {{Bilingual Word Embeddings from Parallel and Non-parallel Corpora for Cross-Language Text Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
}
Figure