Analogy Training Multilingual Encoders

Nicolas Garneau, Mareike Hartmann, Anders Sandholm, Sebastian Ruder, Ivan Vulic, Anders SøgaardAAAI 2021 · OpenReview

TL;DR

This paper tackles the problem of "global inconsistency" in language models, where word meanings are locally related but globally jumbled, hindering cross-lingual understanding. They introduce "WiQueen," a large multilingual analogy dataset, and a Siamese BERT architecture to train models on these analogies, significantly improving global consistency and downstream task performance.

What Problem Does This Paper Address?

Modern language models, especially multilingual ones like mBERT, are fantastic at understanding local word relationships. However, they suffer from "global inconsistency." Imagine a map where all the cities are in the right order relative to their immediate neighbors, but the overall continents are in the wrong places. This is analogous to how these models represent word meanings: while "king" might be related to "man" and "queen," the overall structure of the meaning space across different languages can be distorted. This inconsistency is a major bottleneck for tasks requiring robust cross-lingual transfer, like translation or cross-lingual information retrieval, especially for less-resourced languages. Existing methods to fix this are often limited to English or small datasets.

Methods

The authors propose two main contributions:

  1. WiQueen Dataset: They created a large-scale, multilingual dataset of multi-word analogies (e.g., "Hefei is to Anhui as Guiyang is to Guizhou") derived from Wikidata. This dataset spans 11 languages and covers a wide range of semantic relationships, aiming to provide a more comprehensive resource than existing English-centric or smaller datasets. They also created a "WiQueen+" version that includes entity aliases and descriptions for richer context.
  2. Siamese BERT Architecture for Analogy Training: To address global inconsistency, they fine-tune mBERT using a Siamese network architecture. This architecture processes the four elements of an analogy (w1, w2, w3, w4) through separate BERT encoders. The core idea is to train the model to minimize the distance between the vector representing the analogy's "result" (v1 - v2 + v4) and the vector of the correct target word (v3). They experiment with both Mean Squared Error (MSE) and a contrastive loss, finding the latter to be more effective. This process aims to "ground" the multilingual embeddings in analogy relations, forcing them to be globally consistent.

Datasets

  • WiQueen: A newly introduced, large-scale, multilingual (11 languages) multi-word analogy dataset extracted from Wikidata. It contains 78,000 analogies. A "WiQueen+" version incorporates entity aliases and descriptions.
  • Wikidata: The source for the WiQueen dataset.
  • BATS (Bigger Analogy Test Set): Used for diagnostic experiments on English data.
  • Tatoeba: Used for sentence retrieval evaluation.
  • BUCC2018: Used for sentence retrieval evaluation.
  • MUSE dictionaries: Used for bilingual dictionary induction evaluation.

Key Insights and Ideas

  • Global Inconsistency is Pervasive: The paper empirically demonstrates that even powerful pre-trained multilingual models like mBERT exhibit significant global inconsistency, with analogy retrieval performance degrading sharply as the "distance" of the analogy increases in the embedding space. This holds true across multiple languages.
  • Analogy Training as a Fix: The core hypothesis is that training models on analogies can directly improve their global consistency. This is because analogies inherently capture structured semantic relationships that, when consistently encoded, lead to a more coherent and isomorphic embedding space.
  • Multilingual Analogies are Crucial: The creation of WiQueen highlights the need for multilingual resources to address global inconsistency in cross-lingual models. Relying solely on English analogies is insufficient.
  • Siamese BERT for Analogy Grounding: The Siamese architecture provides an effective framework for fine-tuning BERT on analogy tasks, allowing it to learn to predict the correct analogy target and thus enforce global consistency.
  • Contextual Augmentation (WiQueen+) Helps: While not always outperforming the base analogy training, augmenting analogies with entity aliases and descriptions (WiQueen+) shows promise, especially in improving global consistency.

Key Results

  • Improved Analogy Retrieval: Analogy training significantly boosts P@1 scores for analogy retrieval, with mBERT-WiQueen showing improvements of up to 0.11 and mBERT-WiQueen+ up to 0.15 compared to the baseline mBERT.
  • Enhanced Global Consistency: Analogy training consistently improves global consistency (measured by ρ) across most languages, with mBERT-WiQueen+ showing the largest gains.
  • Bilingual Dictionary Induction: Analogy training leads to substantial improvements in bilingual dictionary induction precision (P@10), indicating better isomorphism between language-specific embedding spaces. For instance, average P@10 for mBERT-WiQueen+ is 0.2252, compared to 0.1161 for mBERT.
  • Sentence Retrieval: Significant improvements are observed in sentence retrieval tasks (Tatoeba and BUCC2018) for most languages after analogy training, demonstrating the downstream benefits of improved global consistency.

Limitations and Future Work

  • Limitations: The paper notes that while analogy training improves global consistency, it might not fully resolve all issues, especially for very distant languages or complex semantic relations. The performance on sentence retrieval, while improved, still shows variation across languages. The authors also mention that the MUSE benchmark for BDI might not be the optimal evaluation for their specific goals.
  • Future Work: The authors suggest exploring more sophisticated analogy training objectives and architectures. They also hint at the potential for analogy training to improve other cross-lingual tasks and the need for further investigation into the relationship between analogy invariance and embedding space isomorphism. The development of more comprehensive multilingual analogy datasets remains an important area.

Summary updated

Abstract

Language encoders encode words and phrases in ways that capture their local semantic relatedness, but are known to be globally inconsistent. Global inconsistency can seemingly be corrected for, in part, by leveraging signals from knowledge bases, but previous results are partial and limited to monolingual English encoders. We extract a large-scale multilingual, multi-word analogy dataset from Wikidata for diagnosing and correcting for global inconsistencies, and then implement a four-way Siamese BERT architecture for grounding multilingual BERT (mBERT) in Wikidata through analogy training. We show that analogy training not only improves the global consistency of mBERT, as well as the isomorphism of language-specific subspaces, but also leads to consistent gains on downstream tasks such as bilingual dictionary induction and sentence retrieval.
BibTeX
@inproceedings{garneau2021analogy,
  author = {Nicolas Garneau and Mareike Hartmann and Anders Sandholm and Sebastian Ruder and Ivan Vulic and Anders Søgaard},
  title = {{Analogy Training Multilingual Encoders}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=m9Qo-VHQ2Zo},
}
Figure