Abstract
Cross-lingual adaptation aims to learn a prediction model in a label-scarce target language by exploiting labeled data from a label-rich source language. An effective cross-lingual adaptation system can substantially reduce the manual annotation effort required in many natural language processing tasks. In this paper, we propose a new cross-lingual adaptation approach for document classification based on learning cross-lingual discriminative distributed representations of words. Specifically, we propose to maximize the log-likelihood of the documents from both language domains under a cross-lingual log-bilinear document model, while minimizing the prediction log-losses of labeled documents. We conduct extensive experiments on cross-lingual sentiment classification tasks of Amazon product reviews. Our experimental results demonstrate the efficacy of the proposed cross-lingual adaptation approach.
Introduction
A persistent challenge in Natural Language Processing (NLP) is the "feature gap" encountered during cross-lingual text classification. When a sentiment classifier is trained on a label-rich source language (e.g., English), it learns specific vocabulary features (e.g., "excellent" implies positive sentiment). However, these models typically fail when applied to a target language (e.g., German) where the vocabulary is entirely different (e.g., "ausgezeichnet"), despite the semantic equivalence.
Traditional solutions like Machine Translation (MT) or bilingual dictionaries often fall short due to computational costs, reliance on high-quality external tools, or a lack of contextual nuance. This paper introduces CL-RL (Cross-Lingual Representation Learning), a semi-supervised method that bridges this gap by learning shared word embeddings across languages without requiring full document translation.
Methodology: The CL-RL Approach
The core of the proposed method is a Semi-Supervised Cross-Lingual Log-Bilinear Document Model. This model learns a shared mathematical vector space where semantically similar words from different languages are positioned closely together.
Three-Part Embedding Structure
To capture both language-specific and shared features, word representations are decomposed into three components:
- Source-specific ($R_s$): Features unique to the source language.
- Target-specific ($R_t$): Features unique to the target language.
- Common ($R_c$): A shared bridge where cross-lingual knowledge is represented.
Joint Optimization
Unlike task-agnostic embedding methods (e.g., Word2Vec), CL-RL employs a joint optimization strategy. The model simultaneously maximizes the log-likelihood of document structures (unsupervised learning of language "shape") and minimizes classification error on labeled source data (supervised learning). By using a small set of "pivot words"—high-frequency, high-sentiment bilingual pairs—the model aligns the $R_c$ space, ensuring the learned vectors are specifically tuned for sentiment classification.
Experimental Results
The approach was evaluated using the Multilingual Sentiment Classification Dataset, comprising Amazon reviews for Books, DVDs, and Music across English (source), German, French, and Japanese (targets). Each task utilized 2,000 labeled source documents alongside varying amounts of labeled and unlabeled target data.
As shown in the experimental results, CL-RL consistently outperformed the state-of-the-art non-MT approach (CL-SCL) and surpassed the Machine Translation (MT) baseline in 8 out of 9 tasks. Notably, the model demonstrated significant robustness to data scarcity, maintaining high accuracy even with as few as 100 labeled target documents.
Sensitivity and Robustness
Further analysis was conducted to determine the impact of hyperparameter selection, specifically the dimensionality of the learned representation space.
The results indicate that CL-RL is remarkably stable across different dimensions (ranging from 100 to 400). This stability suggests that the model effectively captures the underlying semantic structure without requiring exhaustive hyperparameter tuning for different language pairs.
Conclusion
CL-RL provides an efficient, "economic" alternative to full machine translation for cross-lingual tasks. By leveraging task-specific joint optimization and a shared embedding space, the method successfully transfers sentiment knowledge across diverse language families. While the current model relies on a small set of initial pivot words and a linear log-bilinear structure, it establishes a foundational framework for modern domain adaptation and transfer learning in NLP.
Summary updated