TL;DR
This paper tackles the challenge of deciding whether to translate or transliterate English Named Entities (NEs) into Arabic for Machine Translation (MT). They propose a novel classification framework that significantly improves MT quality by reducing erroneous translations, particularly for NEs that are better transliterated.
What Problem Does This Paper Address?
Machine Translation (MT) systems often struggle with Named Entities (NEs) – proper nouns like names of people, places, and organizations. These NEs are frequently "out-of-vocabulary" (OOV) terms, meaning they don't appear in the MT system's training data. A common issue is deciding whether to translate an NE (e.g., "North" as "الشمالية" meaning "the north direction") or transliterate it (e.g., "North" as "نورث"). Incorrect decisions can lead to nonsensical translations and degrade the overall quality of the MT output. This paper addresses the critical need for an automated system that can intelligently make this translation vs. transliteration decision for English NEs in Arabic MT.
Methods
The authors propose a classification-based framework to automate the translation vs. transliteration decision for English NEs. They frame this as a binary classification problem at the token level. Their approach involves:
- Feature Engineering: They develop a rich set of features, categorized into:
- Token-based features: Properties of the token itself (capitalization, punctuation, POS tags).
- Semantic features: NE category (Person, Location, Organization, Miscellaneous) and markers associated with these categories.
- Contextual features: Information about the surrounding tokens within the NE and the NE's position.
- Classifier Training: They use Support Vector Machines (SVMs) as their classification algorithm, leveraging the LIBSVM package. They train two main classifiers:
- Cnews: Trained on a large dataset of NEs extracted from news-related parallel corpora.
- Cdiverse: Trained on a combination of news NEs and a smaller set of diverse-topic NEs from Wikipedia titles, to assess domain robustness.
- Labeling Strategy: They employ a semi-automatic labeling approach using a transliteration similarity measure (Frscore) and manual correction for ambiguous cases.
Datasets
The paper utilizes and describes the creation of two primary datasets:
- News-related Parallel Corpus Data: A bilingual lexicon of approximately 57,000 named-entity pairs was compiled from a large word-aligned parallel corpus. NEs were tagged using an NE tagger and projected from English to Arabic using word alignments.
- Wikipedia Titles Data: A smaller, 4,000 bilingual entity lexicon was created from Wikipedia article titles, specifically focusing on NEs not present in the parallel corpus to test domain effects.
Key Insights and Ideas
- The "Translation vs. Transliteration" Dilemma is Crucial: Directly translating NEs can lead to severe semantic errors, as exemplified by "North" being translated as a direction rather than a name.
- Context and NE Category are Key Predictors: The paper highlights that semantic features (like the NE category) are more influential than local context for making the correct decision.
- Domain Matters, but a Diverse Classifier is Robust: While a classifier trained on news data performs well on news, a classifier trained on a mix of news and Wikipedia data shows better generalization and improvement, especially on diverse Wikipedia test sets.
- Automated Decision-Making Improves MT Quality: The proposed classifier can significantly reduce erroneous translations, leading to a measurable improvement in MT system performance.
Key Results
- Intrinsic Evaluation: The classifiers achieved promising accuracy, with Cnews reaching up to 90.42% on the news test set and Cdiverse reaching 89.18% on a combined test set.
- Extrinsic Evaluation (MT System):
- In an in-domain (news) MT task, the classifier provided a modest but significant improvement in MT quality (e.g., BLEU score increase from 16.63 to 16.91).
- For a Wikipedia translation task, the classifier was shown to reduce erroneous translation of 60.5% of named entities.
- The classifier's accuracy in the MT evaluation was 89%, consistent with intrinsic evaluations.
- False positives were low (5%), representing about 12.6% of total errors.
Limitations and Future Work
- Limitations:
- The extrinsic evaluation for the diverse domain (Wikipedia) was limited due to the lack of a non-news English-Arabic corpus for training the MT system.
- The performance improvement in the news domain was modest, suggesting that NEs in news corpora might be more frequently seen and thus less problematic for standard MT systems.
- Future Work:
- Expand the framework to new languages, provided sufficient data resources and tools are available.
- Explore additional features to further enhance classifier performance.
- Apply the classifier to MT experiments in domains beyond news.
Summary updated