Expanding the Text Classification Toolbox with Cross-Lingual Embeddings

Meryem M’hamdi, Robert West, Theus Hossmann, Michael Baeriswyl, Claudiu Musat

Abstract

Most work in text classification and Natural Language Processing (NLP) focuses on English or a handful of other languages that have text corpora of hundreds of millions of words. This is creating a new version of the digital divide: the artificial intelligence (AI) divide. Transfer-based approaches, such as Cross-Lingual Text Classification (CLTC) - the task of categorizing texts written in different languages into a common taxonomy, are a promising solution to the emerging AI divide. Recent work on CLTC has focused on demonstrating the benefits of using bilingual word embeddings as features, relegating the CLTC problem to a mere benchmark based on a simple averaged perceptron. In this paper, we explore more extensively and systematically two flavors of the CLTC problem: news topic classification and textual churn intent detection (TCID) in social media. In particular, we test the hypothesis that embeddings with context are more effective, by multi-tasking the learning of multilingual word embeddings and text classification; we explore neural architectures for CLTC; and we move from bi- to multi-lingual word embeddings. For all architectures, types of word embeddings and datasets, we notice a consistent gain trend in favor of multilingual joint training, especially for low-resourced languages.

Introduction: Bridging the AI Divide

The rapid advancement of Natural Language Processing (NLP) has primarily benefited high-resource languages like English, leaving behind thousands of others. This disparity creates an "AI divide," where the quality of automated services—from sentiment analysis to topic categorization—varies wildly based on the language spoken. Cross-Lingual Text Classification (CLTC) offers a path forward by allowing models trained on one language (or multiple) to generalize to others within a shared taxonomy.

Historically, CLTC has been treated as a secondary task used to validate word embeddings. This research shifts the focus, treating CLTC as a primary objective. By systematically comparing neural architectures and moving from bilingual to multilingual frameworks, the authors demonstrate how joint training can uplift performance for languages with limited data.

Neural Architectures for CLTC

The study evaluates three core neural architectures, each offering different strengths in processing textual data:

  1. FT-MLP (Averaged Multi-Layer Perceptron): An extension of traditional baseline models that uses an average pooling layer over word embeddings.
  2. MF-CNN (Multi-Filter CNN): Utilizes multiple convolutional filters of varying sizes to capture local n-gram patterns, followed by max-pooling.
  3. bi-GRU-Att (Bidirectional GRU with Attention): Employs recurrent units to capture sequential dependencies in both directions, with an attention mechanism to weigh the importance of specific words.
Figure 1: Neural architectures evaluated for CLTC: (a) FT-MLP, (b) MF-CNN, and (c) bi-GRU-Att.

Beyond these, the researchers introduced a sophisticated multi-tasking model: HAN+Sent-Ali. This hierarchical architecture simultaneously learns to align sentences across languages (minimizing Euclidean distance between parallel translations) and classify documents. This dual-objective approach ensures that the underlying word embeddings are optimized for both semantic alignment and task-specific discriminative power.

Experimental Setup and Datasets

To test these models, the researchers used two distinct datasets:

  • Reuters RCV1/RCV2: A standard benchmark for news topic classification across English (EN), German (DE), French (FR), and Italian (IT).
  • Textual Churn Intent Detection (TCID): A domain-specific dataset from Twitter, where the goal is to identify users intending to switch mobile providers. This dataset is particularly challenging due to the informal nature of social media text.

The study compared "monolingual" training (training and testing on the same language) against "multilingual" training (training on all available languages simultaneously).

Key Results and the Multilingual Advantage

The results across both news and churn datasets reveal a consistent "multilingual advantage." Training on a pool of languages allows the model to learn universal semantic features, which is especially beneficial for languages with smaller training sets.

In the news classification task (CLDC), the hierarchical multi-tasking model (HAN+Sent-Ali) showed dramatic improvements. For English, in a low-data regime of 10k instances, it achieved a 21.4% gain in F1-score compared to simpler models.

Figure 5: Performance comparison on the news classification task (CLDC) across four languages, showing gains from multilingual training.

As shown in the results table, the "gain" metric—representing the improvement of multilingual training over monolingual baselines—is most pronounced for lower-resource languages. For instance, Italian (IT) saw an average F1-score gain of 10.55% with the HAN+Sent-Ali model. Even with the simpler FT-MLP architecture, Italian performance jumped by 7.66%.

In the social media domain (TCID), the gains were even more significant. The average F1-score improvement across architectures ranged from 7.01% to 16.13%. This suggests that the shared semantic space created by multilingual embeddings is robust enough to handle the noise and linguistic variation found in social media.

Conclusion

This systematic exploration of CLTC demonstrates that the "AI divide" can be mitigated through strategic architectural choices and multilingual training. By treating language alignment and text classification as joint tasks, models can effectively leverage data from high-resource languages to boost performance in low-resource ones. While complex models like HAN+Sent-Ali offer the highest gains, they come with increased computational costs. Future research will likely focus on scaling these multi-tasking approaches to even more diverse language families and larger, more imbalanced datasets.

Summary updated

Abstract

Most work in text classification and Natural Language Processing (NLP) focuses on English or a handful of other languages that have text corpora of hundreds of millions of words. This is creating a new version of the digital divide: the artificial intelligence (AI) divide. Transfer-based approaches, such as Cross-Lingual Text Classification (CLTC) - the task of categorizing texts written in different languages into a common taxonomy, are a promising solution to the emerging AI divide. Recent work on CLTC has focused on demonstrating the benefits of using bilingual word embeddings as features, relegating the CLTC problem to a mere benchmark based on a simple averaged perceptron. In this paper, we explore more extensively and systematically two flavors of the CLTC problem: news topic classification and textual churn intent detection (TCID) in social media. In particular, we test the hypothesis that embeddings with context are more effective, by multi-tasking the learning of multilingual word embeddings and text classification; we explore neural architectures for CLTC; and we move from bi- to multi-lingual word embeddings. For all architectures, types of word embeddings and datasets, we notice a consistent gain trend in favor of multilingual joint training, especially for low-resourced languages.
BibTeX
@misc{mhamdi2019expanding,
  author = {Meryem M’hamdi and Robert West and Theus Hossmann and Michael Baeriswyl and Claudiu Musat},
  title = {{Expanding the Text Classification Toolbox with Cross-Lingual Embeddings}},
  year = {2019},
  eprint = {1903.09878},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.09878},
}
Figure