Emu: Enhancing Multilingual Sentence Embeddings with Semantic Specialization

Wataru Hirota, Yoshihiko Suhara, Behzad Golshan, Wang-Chiew Tan

Abstract

We present Emu, a system that semantically enhances multilingual sentence embeddings. Our framework fine-tunes pre-trained multilingual sentence embeddings using two main components: a semantic classifier and a language discriminator. The semantic classifier improves the semantic similarity of related sentences, whereas the language discriminator enhances the multilinguality of the embeddings via multilingual adversarial training. Our experimental results based on several language pairs show that our specialized embeddings outperform the state-of-the-art multilingual sentence embedding model on the task of cross-lingual intent classification using only monolingual labeled data.

Introduction

Multilingual sentence embeddings, such as Facebook’s LASER, are designed to map sentences from diverse languages into a shared vector space. In an ideal system, semantically identical sentences like "What time is the pool open?" (English) and "Wann öffnet der Pool?" (German) would reside in close proximity. However, current models often suffer from textual similarity bias, where embeddings cluster based on surface-level word overlap rather than underlying intent. This bias significantly hinders "zero-shot" cross-lingual transfer, where a model trained on one language (e.g., English) is expected to generalize to others (e.g., Japanese or Spanish) without additional target-language labels.

The Emu Framework

Emu is a lightweight fine-tuning approach designed to specialize base multilingual encoders. The framework introduces two primary components to refine the embedding space:

  1. Semantic Classifier ($C$): This component enforces intent-based grouping. To optimize the geometry of the embedding space, Emu utilizes $L_2$-constrained Softmax Loss, which maps vectors onto a hypersphere to align Euclidean distance with cosine similarity. Additionally, Center Loss is employed to pull embeddings of the same intent toward a central centroid, creating tighter and more distinct clusters.
  2. Language Discriminator ($D$): Fine-tuning on monolingual data often risks "language collapse," where the model loses its multilingual alignment. Emu employs adversarial training where a discriminator attempts to identify the source language of an embedding. The encoder is trained to maximize the discriminator's uncertainty, ensuring the resulting representations remain language-agnostic.
The Emu architecture featuring a multilingual encoder, semantic classifier, and adversarial language discriminator.

Crucially, Emu only requires monolingual labeled data and unlabeled data in target languages, making it highly efficient for domains where parallel corpora are scarce.

Experimental Results

The authors evaluated Emu across three intent-classification datasets (HotelQA, ATIS, and Quora) translated into five languages. Emu consistently outperformed the state-of-the-art LASER model and traditional Machine Translation (MT) pipelines:

  • HotelQA: Emu achieved an accuracy improvement of up to 47.7% over the original LASER model.
  • ATIS: In specific cross-lingual tasks, Emu outperformed the MT + sent2vec baseline by 86.2%.

Qualitative analysis via t-SNE plots demonstrates the impact of this specialization. While original LASER clusters can appear fragmented or "messy," Emu produces distinct, tight clusters where sentences with the same intent from different languages occupy the same region of the vector space.

t-SNE visualizations comparing the embedding spaces of the base model versus Emu-specialized embeddings.

Conclusion

Emu demonstrates that semantic specialization can significantly enhance the utility of multilingual embeddings for downstream tasks like intent classification. By prioritizing semantic intent over textual overlap and maintaining language agnosticism through adversarial training, Emu provides a robust path for zero-shot cross-lingual transfer. Future work may involve applying this framework to larger transformer-based models like mBERT or XLM-R and exploring its efficacy in complex tasks such as cross-lingual document retrieval.

Summary updated

Abstract

We present Emu, a system that semantically enhances multilingual sentence embeddings. Our framework fine-tunes pre-trained multilingual sentence embeddings using two main components: a semantic classifier and a language discriminator. The semantic classifier improves the semantic similarity of related sentences, whereas the language discriminator enhances the multilinguality of the embeddings via multilingual adversarial training. Our experimental results based on several language pairs show that our specialized embeddings outperform the state-of-the-art multilingual sentence embedding model on the task of cross-lingual intent classification using only monolingual labeled data.
BibTeX
@inproceedings{hirota2020emu,
  author = {Wataru Hirota and Yoshihiko Suhara and Behzad Golshan and Wang-Chiew Tan},
  title = {{Emu: Enhancing Multilingual Sentence Embeddings with Semantic Specialization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=O6OuGIZ4e27},
}
Figure