Abstract
We present a new language pair agnostic approach to inducing bilingual vector spaces from non-parallel data without any other resource in a bootstrapping fashion. The paper systematically introduces and describes all key elements of the bootstrapping procedure: (1) starting point or seed lexicon, (2) the confidence estimation and selection of new dimensions of the space, and (3) convergence. We test the quality of the induced bilingual vector spaces, and analyze the influence of the different components of the bootstrapping approach in the task of bilingual lexicon extraction (BLE) for two language pairs. Results reveal that, contrary to conclusions from prior work, the seeding of the bootstrapping process has a heavy impact on the quality of the learned lexicons. We also show that our approach outperforms the best performing fully corpus-based BLE methods on these test sets.
Introduction
Inducing bilingual dictionaries from non-parallel corpora is a fundamental challenge in Natural Language Processing, often characterized as a "chicken and egg" problem. Traditional methods for cross-lingual tasks typically rely on parallel corpora—exact translations that are scarce for most language pairs. While comparable corpora (e.g., Wikipedia articles on similar topics) offer a more abundant alternative, they usually require an initial seed dictionary to bridge the languages. This study proposes a robust bootstrapping method that builds a shared bilingual vector space using only raw, non-parallel text, effectively starting from scratch.
Methodology: The Bootstrapping Framework
The researchers developed a language-agnostic iterative framework designed to map two different languages into a single shared mathematical space. The process follows five key stages:
- Initialization (The Seed): Instead of a human-curated dictionary, a Multilingual Probabilistic Topic Model (MuPTM) is used to identify word pairs that likely share meanings based on document-level thematic similarities.
- Vector Space Construction: Words are represented as high-dimensional vectors based on their local linguistic contexts.
- The Similarity Bridge: The initial seed words serve as shared dimensions, allowing for the direct comparison of vectors across languages.
- Iterative Growth: The model identifies new word pairs that exhibit "symmetry" (where word A is the best translation for B, and vice versa). These high-confidence pairs are added as new dimensions to the vector space.
- Convergence: The cycle repeats until no further reliable dimensions can be identified.
A critical constraint in this framework is the one-to-one translation assumption, which stabilizes the space and prevents semantic drift during the iterative process.
Experimental Setup
The approach was evaluated using Wikipedia article pairs for two language sets:
- Spanish-English (Wiki-ES-EN): Approximately 13,700 article pairs.
- Italian-English (Wiki-IT-EN): Approximately 18,900 article pairs.
Performance was measured against a gold-standard set of 1,000 translation pairs for each language pair.
Key Results and Analysis
The bootstrapping approach consistently outperformed the initial topic model baselines. As shown in the following figure, accuracy improves significantly over iterations as the shared vector space expands.
Accuracy Gains
For the Spanish-to-English task, starting with a seed of 200 pairs, Top-1 accuracy (finding the exact translation) rose from 27.4% to 48.8% upon convergence. Similar trends were observed for the Italian-to-English pair. The method also surpassed state-of-the-art fully corpus-based methods, such as ResponseBC.
The Impact of Seeding
Contrary to previous assumptions that bootstrapping could overcome poor initialization, this study demonstrates that the quality and frequency of the initial seed lexicon are paramount. High-frequency words provide more robust statistical evidence, creating a more stable foundation for the vector space than rare words or simple cognates.
Dimensionality and Convergence
The process typically reaches convergence within 10 to 15 iterations. During this time, the shared space expands from a few hundred initial dimensions to nearly 2,000 reliable dimensions, as illustrated in the growth curves of the experimental results. This expansion is crucial for capturing the semantic nuances required for accurate bilingual lexicon extraction.
Discussion and Conclusion
While the proposed method is highly effective, it currently operates under a one-to-one translation constraint. Future work could explore relaxing this assumption to accommodate the inherent polysemy and complex translation relationships found in natural languages. Additionally, while the method is computationally parallelizable, the sheer volume of word-to-word comparisons remains a bottleneck for scaling to massive vocabularies. Finally, testing this approach on linguistically distant pairs (e.g., English and Arabic) will be essential to confirm its universal applicability.
In summary, this research proves that high-quality bilingual lexicons can be induced from non-parallel data alone, provided the bootstrapping process is anchored by a strong initial seed and refined through symmetric translation filtering.
Summary updated