TL;DR
This paper presents a novel method for machine transliteration (converting names/terms between languages with different writing systems) that doesn't require parallel data. By framing transliteration as a "decipherment" problem and leveraging monolingual resources, they learn phoneme mappings, achieving competitive results on Japanese-to-English name transliteration.
What Problem Does This Paper Address?
Transliteration is crucial for handling names and foreign terms across languages, especially those with different scripts (like Japanese and English). Traditionally, building transliteration systems requires large amounts of parallel data – pairs of names/terms already transliterated. This is expensive and time-consuming to create. The challenge this paper tackles is how to build effective transliteration systems when such parallel data is scarce or unavailable, relying instead on readily available monolingual text.
Methods
The core idea is to treat transliteration as a decipherment problem, akin to breaking a code. They focus on back-transliteration from Japanese katakana to English. Their approach involves learning phoneme substitution tables (mapping English phonemes to Japanese phonemes and vice-versa) using only monolingual data.
Here's a breakdown of their method:
- Framing as Decipherment: They view Japanese katakana as "coded" English. The goal is to "decode" it back to English phonemes.
- Monolingual Resources: They utilize:
- A large corpus of Japanese katakana sequences.
- The CMU pronunciation dictionary for English phonemes.
- The English Gigaword corpus.
- Weighted Finite-State Transducers (WFSTs): They employ a cascade of WFSTs, similar to previous work, but crucially, they focus on learning the WFST
C(phoneme-to-phoneme mapping) without parallel data. - EM Algorithm for Learning: They use the Expectation-Maximization (EM) algorithm to iteratively learn the phoneme substitution probabilities. The objective is to maximize the likelihood of observing the Japanese data, given an English phoneme model and the unknown phoneme mapping.
- Improving the English Model (P(e)): They experiment with different models for English phoneme sequences:
- N-gram models: Starting with 2-gram and then 3-gram models.
- Word-based model: A more sophisticated approach where the English phoneme model is derived from actual English words, ensuring that deciphered sequences are valid English words. This proved to be a significant improvement.
- Consonant Parity Constraint: To make the learning more robust, they introduce a constraint that the number of English and Japanese consonant phonemes in a mapping should be the same. This is based on an observed pattern in transliteration.
- Initialization Strategies: They explore different ways to initialize the WFST
Cbefore EM training, finding that initializing with higher weights for mappings with matching consonant sounds improves results. - Re-ranking with Web Counts: As a final step, they use web search engine counts to re-rank the top transliteration candidates, further improving accuracy.
Datasets
- Japanese Katakana Sequences: 43,717 unique sequences from web newspaper data, reduced to 9,350 unique sequences after splitting multi-word phrases. They also included Japanese versions of the 2008 U.S. Senate roster.
- English Phoneme Data:
- CMU Pronunciation Dictionary (112,151 entries).
- English Gigaword corpus.
- Parallel Data (for comparison): A parallel corpus of 2,683 phoneme sequence pairs was used for baseline comparisons.
Key Insights and Ideas
- Transliteration as Decipherment: This reframing is powerful. By treating it as a code-breaking problem, they can leverage techniques from unsupervised learning and cryptanalysis, bypassing the need for parallel data.
- Monolingual Data Sufficiency: The paper demonstrates that with clever techniques, high-quality transliteration mappings can be learned using only monolingual resources, which are far more abundant than parallel corpora.
- The Importance of a "Valid English" Model: The breakthrough came with using a word-based English phoneme model (P(e)). This ensures that the deciphered output is not just a sequence of phonemes, but one that forms actual English words, significantly improving accuracy.
- Consonant Parity as a Useful Heuristic: The consonant parity constraint, a simple rule derived from observation, acts as a valuable regularization technique, guiding the EM algorithm towards better solutions.
- Web Counts for Re-ranking: Leveraging readily available web search data as a post-processing step is a practical and effective way to boost performance, especially for ambiguous cases.
Key Results
- Competitive Performance without Parallel Data: Their best system, using a word-based English model and consonant parity, achieved a 73% whole-name error rate and 54.2 normalized word edit distance on the U.S. Senator name transliteration task. This is remarkably close to the performance of systems trained with parallel data (40% whole-name error, 25.9 normalized edit distance).
- Improvements with Better English Models: Moving from a 2-gram to a 3-gram English model reduced normalized edit distance from 100.0 to 89.8. The word-based model further reduced it to 57.2.
- Re-ranking Boost: Using web counts for re-ranking improved results from 66% whole-name error to 61%, and normalized edit distance from 49.3 to 48.8.
- Comparable Corpora vs. Non-Parallel: Even using comparable corpora (which have some alignment) yielded a 59% whole-name error, while their best non-parallel system achieved 66%, highlighting the strength of their unsupervised approach.
Limitations and Future Work
- Limitations:
- The paper acknowledges that their best non-parallel system still has a higher error rate than the parallel-trained system.
- The consonant parity constraint is a heuristic and might not hold for all language pairs.
- The "S -> s u" vs. "S -> s" example shows that mappings are not always perfect.
- Future Work:
- Develop more powerful decipherment models and techniques.
- Harness information from a wider variety of monolingual resources.
- Further narrow the gap between parallel-trained and non-parallel-trained approaches.
- Extend the approach to other language pairs beyond Japanese-English.
Summary updated