MultiMirror: Neural Cross-lingual Word Alignment for Multilingual Word Sense Disambiguation

Luigi Procopio, Edoardo Barba, Federico Martelli, Roberto NavigliIJCAI 2021 · DOI

TL;DR

This paper introduces MULTIMIRROR, a novel approach for multilingual Word Sense Disambiguation (WSD) that leverages neural cross-lingual word alignment. By projecting sense annotations across parallel sentences, MULTIMIRROR significantly improves WSD performance, achieving state-of-the-art results on established benchmarks, especially for distant languages like Japanese.

What Problem Does This Paper Address?

Word Sense Disambiguation (WSD) aims to identify the correct meaning of a word in a given context. While neural models have achieved impressive performance in English, scaling WSD to other languages is hindered by the scarcity of high-quality, sense-annotated data. This "knowledge acquisition bottleneck" is particularly challenging for low-resource languages. Existing approaches like knowledge bases or annotation projection have limitations, often requiring extensive external resources or struggling with cross-lingual transfer. This paper addresses the need for an efficient and effective method to create multilingual WSD datasets by leveraging parallel sentences and advanced word alignment techniques.

Methods

The core of MULTIMIRROR is a novel neural architecture for cross-lingual word alignment. This model takes a pair of parallel sentences as input and jointly aligns all source and target tokens simultaneously. It utilizes a multilingual BERT (mBERT) model to obtain contextualized token representations, followed by a token-level Transformer Encoder for further contextualization. The alignment probabilities are then computed using a series of linear and ReLU layers.

Once the word alignment matrix is generated, MULTIMIRROR projects sense annotations from a source language (typically English, with existing WordNet senses) to a target language using parallel sentences. The projection process involves identifying corresponding spans in the target sentence based on the alignment matrix and then assigning the sense tag. This method is designed to be efficient, requiring only a few hundred training sentences for the alignment model.

Datasets

The paper introduces four new manually-annotated datasets for word alignment in the following language pairs: English-French (En-Fr), English-German (De-En), English-Italian (It-En), and English-Spanish (Es-En). These datasets were created by manually aligning 300 sentence pairs from WikiMatrix for Italian and Spanish, and using existing datasets for the other pairs. For WSD evaluation, they use SemEval-13, SemEval-15, and the XL-WSD dataset for Japanese. They also generate target corpora by automatically translating English WSD datasets (SemCor and WNG) using a multilingual translation model.

Key Insights and Ideas

  • Parallel sentences are a natural setting for annotation projection: The authors argue that projecting senses across parallel sentences is a more direct and effective approach than transferring annotations between different types of text.
  • Neural word alignment is key to efficient cross-lingual transfer: Their novel neural alignment model, trained on limited data, can effectively align tokens across languages, forming the backbone for sense projection.
  • Projecting senses from English is a viable strategy: By leveraging the strong performance of English WSD datasets and projecting them to other languages, they demonstrate a path towards multilingual WSD without requiring extensive manual annotation in each target language.
  • Distant languages benefit significantly: The approach shows particularly strong improvements for languages like Japanese, which are linguistically distant from English, suggesting its robustness in challenging cross-lingual scenarios.

Key Results

MULTIMIRROR achieves state-of-the-art results on established WSD benchmarks across multiple languages. Specifically:

  • Word Alignment: The proposed neural alignment model outperforms the previous best system (Nagata et al., 2020) by up to 2 F1 points, especially on the Japanese-English pair.
  • WSD Performance: When using the automatically generated sense-tagged corpora with a simple mBERT-based classifier, MULTIMIRROR achieves state-of-the-art performance on SemEval-13, SemEval-15, and XL-WSD (Japanese).
  • Japanese Performance: For Japanese, MULTIMIRRORFT significantly outperforms a strong competitor (MuLaN) and even direct zero-shot approaches, showing a gap of over 4 F1 points compared to Ø-shot-SemCor+WNG.
  • Coverage: MULTIMIRROR demonstrates superior coverage in terms of projected senses and synsets compared to the MuLaN system.

Limitations and Future Work

  • Limitations: The paper acknowledges that their manually annotated alignment datasets, while effective, are smaller than some existing ones. They also note that while their approach is faster than some competitors, the alignment process itself can still be computationally intensive.
  • Future Work: The authors plan to explore the scalability of their framework by considering comparable sentences (instead of strictly parallel ones) for word alignment and sense projection. They also suggest investigating the impact of different source language annotations and further refining the projection heuristics.

Summary updated

BibTeX
@inproceedings{procopio2021multimirror,
  author = {Luigi Procopio and Edoardo Barba and Federico Martelli and Roberto Navigli},
  title = {{MultiMirror: Neural Cross-lingual Word Alignment for Multilingual Word Sense Disambiguation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/539},
}
Figure