Abstract
This paper introduces a new method for identifying named-entity (NE) transliterations within bilingual corpora. Current state-of-the-art approaches usually require annotated data and relevant linguistic knowledge which may not be available for all languages. We show how to effectively train an accurate transliteration classifier using very little data, obtained automatically. To perform this task, we introduce a new active sampling paradigm for guiding and adapting the sample selection process. We also investigate how to improve the classifier by identifying repeated patterns in the training data. We evaluated our approach using English, Russian and Hebrew corpora.
Introduction
Named Entity (NE) transliteration—the phonetic conversion of names across different scripts—is a critical component for Machine Translation and Information Retrieval. Names often fall outside standard vocabularies, making them difficult to translate without specialized phonetic mappings. Traditional supervised methods require large, manually annotated datasets, which are scarce for many language pairs. This research addresses the challenge of training high-quality transliteration classifiers with minimal data and linguistic knowledge.
Methodology
The proposed framework, termed Sufficient Spanning Features (SSF), relies on three pillars: active sampling, automated labeling, and feature weighting.
Active Sampling Paradigm
Instead of random selection, the system employs a "directed" sampling approach. It maintains a large reference set of names in the source language and analyzes their character n-gram distributions. The goal is to select training samples that minimize the L1 distance between the training set's distribution and the overall reference distribution. This ensures the training data "spans" the phonetic variety of the language efficiently.
Transliteration Oracle and Feature Weighting
To avoid manual annotation, the system uses Wikipedia as an "oracle." By leveraging inter-language links between articles (e.g., English and Russian pages for the same entity), it automatically extracts correct transliteration pairs.
Once a small set of positive pairs is obtained, the model calculates "activation strengths" for character-pair features. This weighting mechanism helps a regularized perceptron classifier focus on the most reliable phonetic correlations found in the data.
Experimental Setup
The approach was evaluated on two primary datasets:
- English-Russian News Corpus: A benchmark for comparison with previous state-of-the-art models (e.g., Klementiev and Roth, 2006).
- Wikipedia-derived Corpora: A reference set of 50,000 English names used to generate training pairs for English-Russian and English-Hebrew pairs.
Results and Discussion
The core finding is that the quality of training data is more significant than its quantity. The SSF directed sampling method achieved superior recall compared to random sampling, even when the random set was twice as large.
On the English-Russian task, the SSF method outperformed the 2006 baseline, achieving a 0.68 Top-1 Recall with only 200 directed samples, compared to 0.63 for the baseline. The method also proved robust for Hebrew, where feature weighting significantly boosted Top-1 accuracy from 0.37 to 0.52.
Conclusion
This research demonstrates that active sample selection based on distribution matching can drastically reduce the data requirements for building accurate transliteration models. By utilizing Wikipedia as an automated oracle and applying intelligent feature weighting, the system achieves state-of-the-art performance with minimal human intervention, offering a scalable solution for low-resource language pairs.
Summary updated