Abstract
Phrase-based Statistical Machine Translation (SMT) often struggles with highly inflected languages due to data sparsity. This paper presents a method to handle verb phrase morphology in English-to-Hindi translation by augmenting the phrase table with generated verb forms. By leveraging linguistic templates and monolingual data, the authors achieve significant improvements in translation quality, with gains of up to 4.8 BLEU points.
Introduction: The Challenge of Inflection
Standard SMT models rely on observing word sequences in parallel corpora. However, in highly inflected languages like Hindi, a single verb root can manifest in dozens of forms based on gender, number, tense, and aspect. It is statistically improbable for a standard parallel corpus to contain every possible variation. When an MT system encounters an unseen verb form, it typically fails or produces a poor translation. This "data sparsity" is a major bottleneck for low-resource or highly inflected languages.
Methodology
The researchers proposed two primary strategies to enrich the MT system's phrase table with missing verb forms:
1. Manual Template-based Generation
This approach utilizes the linguistic structure of Hindi, where complex actions often involve "light verbs" (e.g., karna [to do], lena [to take]).
- Clustering: Verbs are grouped into classes based on their auxiliary or light verb.
- Templates: Generic templates are created (e.g.,
<verb> raha thafor "was <verb>+ing"). - Human-in-the-loop: A human translates a single representative pair for each class. The system then automatically populates these templates with other verbs from the same class, generating thousands of new entries.
2. Automatic Generation
This method relies on large-scale monolingual data to identify verb variations. Word alignment models (such as GIZA++) are used to learn mappings between source-side and target-side phrases, creating generalized templates without manual intervention.
The generated phrases are injected directly into the phrase table, followed by a re-normalization of probabilities. This ensures that the model's existing knowledge remains intact while specifically addressing the gaps in verb morphology.
Experimental Setup
The study utilized the following datasets:
- Parallel Corpus: 280,000 English-Hindi sentence pairs for baseline training.
- Monolingual Data: 6 million English and 1.4 million Hindi sentences for discovering verb variations.
- Test Sets: An in-domain set of 4,000 sentences and an out-of-domain news set of 715 sentences.
Key Results
The proposed methods yielded substantial improvements in both translation quality and coverage.
- BLEU Score Gains: The manual generation method provided a massive 4.8 BLEU point improvement on in-domain data and a 1.1 point gain on out-of-domain data.
- Coverage Improvement: The system's ability to recognize and translate long verb phrases (length 4) nearly doubled, jumping from 40% to approximately 80%.
As shown in the results, the "Base+ManVPPT" (Manual Verb Phrase Phrase Table) configuration consistently outperformed the baseline across various training data sizes, demonstrating the robustness of the approach even as the amount of parallel data increases.
Conclusion and Future Work
By targeting the specific linguistic challenge of verb morphology, the authors demonstrated that even small amounts of human-guided template generation can significantly outperform standard data-heavy approaches.
Future work aims to address long-range dependencies, where verb inflection depends on a distant subject, and contextual ambiguity (polysemy). Additionally, the researchers hope to develop more language-independent tools for extracting verb phrases to extend this methodology to other highly inflected languages.
Summary updated