TL;DR
This paper presents a novel system for automatically building multilingual dictionaries, particularly for "resource-poor" languages, by leveraging parallel texts like the Bible. The core innovation is a method to combine multiple pairwise translation models into a single, more robust multilingual model, significantly improving dictionary quality compared to existing tools.
What Problem Does This Paper Address?
Modern natural language processing (NLP) systems, especially those involved in machine translation, thrive on large amounts of annotated data. However, this data is scarce for most of the world's languages, which are often referred to as "resource-poor." This lack of data creates a significant barrier to developing NLP tools for these languages, even though many are spoken by millions of people. The authors aim to bridge this gap by creating a system that can induce multilingual dictionaries from parallel texts, even when only a single, shared multitext (like the Bible) is available across related languages. This is crucial because dictionaries are fundamental building blocks for many NLP tasks.
Methods
The paper proposes a system that builds a multilingual dictionary by learning a joint probability distribution over words in multiple related languages. The core idea is to combine multiple pairwise translation models.
- Multilingual Dictionary Induction: The system assumes the availability of a parallel multitext (e.g., the Bible) aligned across several related languages ($L_1, \dots, L_n$). The goal is to learn the joint distribution $P(x_1, \dots, x_n)$ where $x_i$ is a word from language $L_i$.
- Combining Pairwise Models: Instead of directly learning the joint distribution, the system first learns pairwise models $P(x_i | x_j)$ and then combines them. This is achieved by minimizing a distance measure (specifically, a generalization of the Bhattacharyya distance) between the combined model and the individual pairwise models. This process involves an iterative optimization algorithm.
- Feature-Rich Pairwise Models: To build the initial pairwise translation models $P_{A \to B}(y|x)$, the authors propose a mixture model that incorporates several features capturing relationships between words in related languages:
- GIZA++ (Forward and Backward): Standard statistical machine translation alignment tools are used to build initial word translation probabilities.
- Character-based Model: This model leverages the fact that related languages often share phonetic or orthographic similarities. It models word translation as a sequence of character translations, using GIZA++ on character-level "bitexts."
- Prefix and Suffix Models: These models capture morphological similarities by grouping words based on shared prefixes or suffixes, assuming that words with similar morphology are likely to translate to words with similar morphology.
- Constituency Model: This model attempts to incorporate syntactic information by considering word groupings (e.g., three-word sequences) as potential constituents.
Datasets
- Training Data: The primary dataset used is the Bible, available in parallel across multiple languages. The authors specifically mention using the Russian, Ukrainian, and Belorussian Bibles for evaluation, and later incorporate Polish.
- Evaluation Data: A hand-built Russian-Ukrainian dictionary is used for comparison.
Key Insights and Ideas
- Leveraging Language Relatedness: The core insight is that the relatedness between languages is a powerful signal that can be exploited to build better translation models and dictionaries, even with limited data.
- Combining Multiple Pairwise Models: A significant finding is that combining multiple pairwise models into a single joint model leads to better dictionary quality than relying on individual pairwise models. This is because the relationships between language pairs provide complementary information.
- Feature Engineering for Related Languages: The paper demonstrates that incorporating features like character-level similarity, prefixes, and suffixes significantly improves translation model performance, especially for related languages where these features are more pronounced.
- The Bible as a Universal Multitext: The Bible's availability in numerous languages makes it an ideal, albeit specific, resource for training multilingual systems, especially for languages lacking other parallel corpora.
Key Results
The paper evaluates its system by building a tri-lingual dictionary (Russian-Ukrainian-Belorussian) and comparing the Russian-Ukrainian portion to a hand-built dictionary.
- Significant Improvement over Baseline: The proposed system, even with just the GIZA++ baseline, achieves a 62.3% match rate. By progressively adding components (character, backward, prefix, suffix, constituency), the performance for Russian-Ukrainian translation improves to 84.5% (pair model) and 85.2% (joint model).
- Joint Model Advantage: The joint model, which considers all languages simultaneously, consistently outperforms pairwise models.
- Effectiveness of Features: Adding features like character-based translation and morphological components (prefix/suffix) yields substantial gains.
- Tuning with a Third Language: Using a third language (Belorussian) for tuning parameters proved effective, achieving near-optimal results.
- Best Result: The best result reported is 85.7% for Russian-Ukrainian translation, achieved by marginalizing a tri-lingual joint model and incorporating Polish as a fourth language.
Limitations and Future Work
- Corpus Dependency: The resulting dictionary is limited to words present in the training corpus. Rare, domain-specific, or technical terms might be missing.
- Morphological Models: While prefix and suffix models are included, the authors acknowledge the need for models of infixes for languages like Hebrew or Arabic to achieve greater generality.
- Testing on Other Families: The approach has not yet been extensively tested on language families beyond those explored, and future work includes testing on Romance languages.
- Expanding to General Settings: The ultimate goal is to produce tools like parsers for low-resource languages. This would involve using the built dictionary to improve pairwise models further and potentially expanding dictionary coverage by mining the web for non-parallel texts.
Summary updated