Constructing Transliteration Lexicons from Web Corpora

Jin-Shea Kuo, Ying-Kuei YangACL 2004

TL;DR

This paper presents an automated method for building transliteration lexicons, crucial for machine translation, by leveraging web query logs. Their novel approach bypasses the need for expensive speech recognition data or letter-to-phoneme systems, enabling efficient cross-language term extraction from readily available web text.

What Problem Does This Paper Address?

Machine translation systems often struggle with words not present in their dictionaries, especially proper nouns and newly coined terms. Transliteration, the process of converting a word from one script to another while preserving its pronunciation, is key to handling these "out-of-vocabulary" (OOV) terms. Traditionally, building transliteration lexicons (lists of word pairs) is a manual and labor-intensive process. Existing automated methods often rely on complex, resource-heavy techniques like automated speech recognition (ASR) to generate confusion matrices or require pre-existing letter-to-phoneme (L2P) systems, which are not always available for all language pairs. This paper tackles the challenge of creating these lexicons automatically and efficiently, particularly for language pairs where such resources are scarce.

Methods

The core of the proposed method is a simple syllable alignment algorithm (SSAA). This algorithm is used to automatically construct confusion matrices for cross-language syllable-phoneme conversion. The process involves:

  1. Syllable Alignment: SSAA aligns syllables between source and target language terms. A key assumption for initial alignment is that the source and target terms have an equal number of syllables.
  2. Phoneme Decomposition: Syllables are further decomposed into phonemes.
  3. Cross-Linguistic Relation Construction: Based on these alignments, two types of cross-linguistic relations are constructed:
  4. Cross-language Phoneme-to-Phoneme (CPP): This method uses a phoneme-based syllabification algorithm (PSA) and requires an L2P system for the source language.
  5. Cross-language Text-to-Phoneme (CTP): This method uses a simpler text-based syllabification algorithm (TSA) and does not require an L2P system. This makes it more broadly applicable.
  6. Iterative Extraction: The constructed relations are used to extract transliterated-term pairs from web query logs. The extracted pairs are then used to refine the cross-linguistic mapping, and the process iterates to improve accuracy and extract more pairs.

The paper highlights that the CTP approach, while using a simpler syllabification, offers a significant advantage by not needing an L2P system, making it more practical for many language pairs.

Datasets

  • Training Corpus (SET1): A 500MB English-Chinese text corpus collected from the internet (15,822,984 pages). From this, 80,094 qualifying sentences (5MB) were extracted.
  • Evaluation Corpus (SET2): A larger corpus (3.17GB, 67,944 pages) obtained by iteratively submitting queries to search engines.
  • Initial Seed Lexicon: 200 verified English-Chinese transliterated-term pairs with an equal number of syllables were manually collected to bootstrap the training process.

Key Insights and Ideas

  • Web Query Logs as a Rich Source: The paper demonstrates that Internet search engine query logs are a valuable, underutilized resource for automatically discovering transliterated-term pairs.
  • SSAA's Simplicity and Effectiveness: The SSAA algorithm provides a straightforward way to build cross-linguistic mappings without relying on complex external tools like ASR.
  • CTP's Broad Applicability: The CTP approach, by eschewing the need for an L2P system, significantly lowers the barrier to entry for transliteration lexicon construction for new language pairs.
  • Iterative Refinement is Key: The iterative nature of the extraction process, where newly found pairs improve the mapping for subsequent rounds, is crucial for achieving a large quantity of high-quality transliterated terms.
  • Phoneme vs. Syllable Conversion: While phoneme-based conversion initially yields more pairs, syllable-based conversion (especially with improved syllabification) ultimately outperforms it due to better handling of linguistic structures and a larger effective "vocabulary" of phonetic units.

Key Results

  • The automated approach successfully extracted over 200,000 transliterated-term pairs from web query results using the CPP mapping over 7 days.
  • The CTP mapping extracted 110,295 pairs, while the CPP mapping extracted 201,732 pairs from the same set of internet search engine queries. This indicates CPP's higher yield, likely due to the more sophisticated phoneme-based processing.
  • The paper shows that the CTP approach, despite using a simpler syllabification algorithm (TSA), is effective in automatically extracting a large quantity of terms. The authors suggest that a more refined TSA could further improve CTP's performance.
  • Experimental results indicate that the proposed approach is "very promising" for transliterated-term extraction.

Limitations and Future Work

  • TSA Simplicity: The paper acknowledges that the Text-to-Phoneme (TP) syllabification algorithm (TSA) used in the CTP approach is "very simple and rudimentary." A more sophisticated TSA could lead to better results for CTP.
  • Initial Seed Lexicon: The method still requires an initial set of manually verified transliterated-term pairs to bootstrap the training. While reduced in quantity (200 pairs), it's still a manual step.
  • Future Work: The authors suggest that improving the TSA would enhance the performance of the CTP method. They also imply that the approach can be extended to other language pairs beyond English-Chinese.

Summary updated

BibTeX
@inproceedings{kuo2004constructing,
  author = {Jin-Shea Kuo and Ying-Kuei Yang},
  title = {{Constructing Transliteration Lexicons from Web Corpora}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2004},
}
Figure