Language identification of names with SVMs

Aditya Bhargava, Grzegorz KondrakNAACL 2010

TL;DR

This paper introduces a novel approach to language identification for short texts, specifically proper nouns, using Support Vector Machines (SVMs) with n-gram counts as features. Their method significantly outperforms traditional character-level n-gram language models, demonstrating its effectiveness for challenging tasks like identifying the origin of names.

What Problem Does This Paper Address?

Language identification, the task of determining the language of a given text, has numerous applications in natural language processing, from improving speech recognition to aiding machine transliteration. Traditionally, this has been tackled using character-level n-gram language models. However, these models struggle with short texts, such as names, where the limited character sequence can lead to ambiguous language cues. This paper addresses this limitation by proposing and evaluating an SVM-based approach specifically for identifying the language of names, which are often too short for traditional methods to handle effectively.

Methods

The core of this paper's contribution lies in its proposed method: using Support Vector Machines (SVMs) with character n-gram counts as features for language identification. Instead of relying on full language models, they extract n-gram frequencies (up to a certain length) from a given name and use these counts as input to an SVM classifier. They also include word length as an additional feature. The authors experimented with different SVM kernels (linear, sigmoid, RBF) and found that linear, sigmoid, and RBF kernels performed best. They also explored various preprocessing steps, such as normalizing feature vectors and adjusting n-gram weights, but found these to have negligible impact. For implementation, they utilized the LIBLINEAR package for linear kernels and LIBSVM for RBF and sigmoid kernels.

Datasets

The authors evaluated their approach on two primary datasets:

  • Transfermarkt corpus: This corpus consists of European soccer player names, annotated with one of 13 national languages. It includes separate lists for last names (12,051 names, average length 7.8) and full names (14,914 names, average length 14.8). Diacritics were removed to avoid trivializing the task.
  • CEJ corpus (Chinese-English-Japanese): This corpus is a combined list of first names and surnames, classified as Chinese, English, or Japanese. It contains 97,115 names with an average length of 7.6 characters. This corpus was previously used for semantic transliteration into Chinese.
  • English-Hindi corpus of names: For the machine transliteration application, they used a corpus containing 1000 names in both Latin and Devanagari scripts. These names were manually classified as either Indian or non-Indian origin.

Key Insights and Ideas

  • SVMs excel at short text language identification: The paper strongly demonstrates that SVMs, when used with n-gram features, significantly outperform traditional language models for identifying the language of names. This is a crucial finding, as names are a common type of short text where traditional methods falter.
  • N-gram counts are powerful features for names: The success of the SVM approach hinges on using n-gram counts as features. This suggests that even short sequences of characters carry sufficient linguistic information for language discrimination when processed by a robust classifier.
  • Language identification as a pre-processing step for transliteration: The paper explores whether language identification can improve machine transliteration. While their simple approach of splitting data and training separate models didn't yield significant gains, they hypothesize that the transliteration system might implicitly leverage language origin information, and more sophisticated integration could be beneficial.
  • Data size is critical for transliteration: Their experiments on transliteration suggest that the reduction in training data size due to language identification was a significant factor in the lack of performance improvement. This highlights the trade-off between specialized models and the amount of data available for training.

Key Results

  • Transfermarkt Corpus: SVMs achieved significantly better accuracy than language models. For last names, SVMs yielded an accuracy of around 56.4%, compared to 44.7% for language models. For full names, SVMs reached 79.9% accuracy, while language models achieved 54.2%. The differences between SVM kernels were not statistically significant.
  • CEJ Corpus: SVMs (linear kernel) achieved an overall accuracy of 97.6% for language identification, compared to 94.8% for language models. This represents a reduction in error rate of over 50%.
  • Machine Transliteration: Using language identification to split the English-Hindi corpus and train separate transliteration models resulted in a combined top-1 accuracy of 46.0%. This was not statistically significantly different from a single transliteration model trained on the full data (47.0%). However, a random split of the data yielded a significantly lower accuracy of 41.5%, suggesting that language identification does provide useful information, but the data reduction was detrimental.

Limitations and Future Work

  • Transliteration performance: The authors acknowledge that their simple method of using language identification as a pre-processing step for machine transliteration did not yield significant improvements. They attribute this, in part, to the reduction in training data size for the specialized transliteration models.
  • Future work in transliteration: They propose exploring more advanced methods for incorporating language identification into transliteration, such as using language identification probabilities as features for the transliteration system or splitting the data into non-disjoint sets to allow models to learn from shared information.
  • Generalizability of SVM approach: While the SVM approach shows promise for names, further research could explore its effectiveness on other types of short texts or in scenarios with more languages.

Summary updated

BibTeX
@inproceedings{bhargava2010language,
  author = {Aditya Bhargava and Grzegorz Kondrak},
  title = {{Language identification of names with SVMs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure