Semi-Supervised Representation Learning for Cross-Lingual Text Classification

Min Xiao, Yuhong GuoEMNLP 2013 · OpenReview

Abstract

Cross-lingual adaptation aims to learn a prediction model in a label-scarce target language by exploiting labeled data from a label-rich source language. An effective cross-lingual adaptation system can substantially reduce the manual annotation effort required in many natural language processing tasks. In this paper, we propose a new cross-lingual adaptation approach for document classification based on learning cross-lingual discriminative distributed representations of words. Specifically, we propose to maximize the log-likelihood of the documents from both language domains under a cross-lingual log-bilinear document model, while minimizing the prediction log-losses of labeled documents. We conduct extensive experiments on cross-lingual sentiment classification tasks of Amazon product reviews. Our experimental results demonstrate the efficacy of the proposed cross-lingual adaptation approach.

Introduction

A persistent challenge in Natural Language Processing (NLP) is the "feature gap" encountered during cross-lingual text classification. When a sentiment classifier is trained on a label-rich source language (e.g., English), it learns specific vocabulary features (e.g., "excellent" implies positive sentiment). However, these models typically fail when applied to a target language (e.g., German) where the vocabulary is entirely different (e.g., "ausgezeichnet"), despite the semantic equivalence.

Traditional solutions like Machine Translation (MT) or bilingual dictionaries often fall short due to computational costs, reliance on high-quality external tools, or a lack of contextual nuance. This paper introduces CL-RL (Cross-Lingual Representation Learning), a semi-supervised method that bridges this gap by learning shared word embeddings across languages without requiring full document translation.

Methodology: The CL-RL Approach

The core of the proposed method is a Semi-Supervised Cross-Lingual Log-Bilinear Document Model. This model learns a shared mathematical vector space where semantically similar words from different languages are positioned closely together.

Three-Part Embedding Structure

To capture both language-specific and shared features, word representations are decomposed into three components:

  1. Source-specific ($R_s$): Features unique to the source language.
  2. Target-specific ($R_t$): Features unique to the target language.
  3. Common ($R_c$): A shared bridge where cross-lingual knowledge is represented.

Joint Optimization

Unlike task-agnostic embedding methods (e.g., Word2Vec), CL-RL employs a joint optimization strategy. The model simultaneously maximizes the log-likelihood of document structures (unsupervised learning of language "shape") and minimizes classification error on labeled source data (supervised learning). By using a small set of "pivot words"—high-frequency, high-sentiment bilingual pairs—the model aligns the $R_c$ space, ensuring the learned vectors are specifically tuned for sentiment classification.

Experimental Results

The approach was evaluated using the Multilingual Sentiment Classification Dataset, comprising Amazon reviews for Books, DVDs, and Music across English (source), German, French, and Japanese (targets). Each task utilized 2,000 labeled source documents alongside varying amounts of labeled and unlabeled target data.

Performance comparison of CL-RL against baselines across nine cross-lingual sentiment classification tasks.

As shown in the experimental results, CL-RL consistently outperformed the state-of-the-art non-MT approach (CL-SCL) and surpassed the Machine Translation (MT) baseline in 8 out of 9 tasks. Notably, the model demonstrated significant robustness to data scarcity, maintaining high accuracy even with as few as 100 labeled target documents.

Sensitivity and Robustness

Further analysis was conducted to determine the impact of hyperparameter selection, specifically the dimensionality of the learned representation space.

Accuracy of CL-RL across different embedding dimensions for German, French, and Japanese.

The results indicate that CL-RL is remarkably stable across different dimensions (ranging from 100 to 400). This stability suggests that the model effectively captures the underlying semantic structure without requiring exhaustive hyperparameter tuning for different language pairs.

Conclusion

CL-RL provides an efficient, "economic" alternative to full machine translation for cross-lingual tasks. By leveraging task-specific joint optimization and a shared embedding space, the method successfully transfers sentiment knowledge across diverse language families. While the current model relies on a small set of initial pivot words and a linear log-bilinear structure, it establishes a foundational framework for modern domain adaptation and transfer learning in NLP.

Summary updated

Abstract

Cross-lingual adaptation aims to learn a prediction model in a label-scarce target language by exploiting labeled data from a labelrich source language. An effective crosslingual adaptation system can substantially reduce the manual annotation effort required in many natural language processing tasks. In this paper, we propose a new cross-lingual adaptation approach for document classification based on learning cross-lingual discriminative distributed representations of words. Specifically, we propose to maximize the loglikelihood of the documents from both language domains under a cross-lingual logbilinear document model, while minimizing the prediction log-losses of labeled documents. We conduct extensive experiments on cross-lingual sentiment classification tasks of Amazon product reviews. Our experimental results demonstrate the efficacy of the proposed cross-lingual adaptation approach.
BibTeX
@inproceedings{xiao2013semi,
  author = {Min Xiao and Yuhong Guo},
  title = {{Semi-Supervised Representation Learning for Cross-Lingual Text Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  url = {https://openreview.net/forum?id=BJbQqGGd-H},
}
Figure