TL;DR
This paper challenges the dominance of phrase-based statistical machine translation (SMT) by proposing Minimal Translation Units (MTUs) and integrating them with syntax-aware models. The authors demonstrate that MTUs, combined with a tree-based ordering model, can achieve comparable or even superior translation quality to traditional phrase-based systems, while also enabling the use of smaller, more manageable phrase tables.
What Problem Does This Paper Address?
Statistical Machine Translation (SMT) has seen significant progress, with phrase-based SMT (PBSMT) systems largely outperforming earlier word-based approaches. However, PBSMT faces several theoretical and practical challenges:
- Non-compositionality: While phrases handle idiomatic expressions well, they struggle with truly fixed phrases that don't easily break down.
- Local Re-ordering: Phrases capture local re-ordering but lack a robust strategy for global re-ordering across longer distances.
- Partitioning Limitation: PBSMT partitions sentences into phrases, assuming the partitioning itself is optimal and treating each phrase translation independently. This can lead to biases and fails to model phenomena that span across phrase boundaries, like agreement.
- Model Size and Sparsity: Large phrase tables are necessary for good performance but lead to significant storage and sparsity issues, especially for longer phrases.
Recent work has explored syntax-based SMT and hierarchical phrases to address some of these issues, but problems with estimation and partitioning often remain. This paper aims to address all these challenges simultaneously.
Methods
The authors propose a novel approach that combines two key ideas:
- Minimal Translation Units (MTUs): Inspired by bilingual n-gram channel models, MTUs are defined as smaller units than traditional phrases. They are extracted based on word alignments and adhere to specific constraints: no word alignment links between distinct MTUs, and no smaller MTUs can be extracted without violating these constraints. This allows for more flexible units, including zero-to-one and one-to-zero translations.
- Syntax-Aware Models: The system leverages a syntactically-informed treelet approach (similar to their previous work) and integrates MTUs into a log-linear model. This model incorporates several components:
- MTU-based n-gram models: These models capture local context and interactions between adjacent MTUs, addressing the partitioning problem and providing robust estimation. They are trained using different "walk orders" (source order, target order, and dependency tree order) to capture distinct types of information.
- Tree-based ordering model: This model handles global re-ordering by predicting the head-relative position of nodes in a dependency tree.
- Traditional channel models: MLE and lexical weighting models are used for phrase/treelet translation probabilities.
- Word-based models: Target language models for fluency, word count, and phrase count features.
- Syntactic models: Including a tree-based bigram target language model.
The system uses a dependency tree-based beam search decoder.
Datasets
The experiments were conducted on technical software documentation for two language pairs:
- English to French (EF)
- English to Japanese (EJ)
The paper provides a table (Table 4.1) detailing the characteristics of these datasets, including the number of sentences, words, vocabulary size, and out-of-vocabulary (OOV) rates for training, development, and test sets.
Key Insights and Ideas
- MTUs as a Solution to Phrase-Based Problems: MTUs offer a way to break down translation into smaller, more manageable units, mitigating sparsity and estimation issues associated with large phrases.
- Syntax and MTUs Complement Each Other: Combining syntax-aware models (like treelets and dependency tree ordering) with MTU-based n-gram models proves more effective than either approach alone.
- The Power of Small Phrases: The research suggests that using smaller phrases (or MTUs) can lead to improved translation quality, especially when combined with robust contextual models like MTU n-grams. This challenges the conventional wisdom that larger phrases are always better.
- Orthogonality of Models: The MTU n-gram models appear to capture information somewhat orthogonal to traditional phrasal models, leading to additive improvements when combined.
- Resilience to Phrase Size: The proposed system demonstrates increased resilience to smaller phrase sizes compared to standard phrase-based systems.
Key Results
- The proposed system, using MTU bigram models and a tree-based ordering model (model set (1)-(4)), achieved the highest BLEU scores on both English-French (50.5±2.1) and English-Japanese (36.2±0.9) test sets.
- Augmenting an SMT system with MTU n-gram models improved quality, while replacing standard phrasal channel models with MTU n-gram channel models resulted in similar performance.
- When comparing systems with varying phrase/treelet sizes, the MTU-based system showed a significant advantage with smaller units. For instance, at size 2, the MTU system (47.7±2.1 EF, 33.8±0.8 EJ) outperformed the phrasal decoder (40.4±1.9 EF, 29.7±0.7 EJ).
- The paper notes that the source order MTU n-gram model had a major contribution to the overall score, suggesting potential redundancy among the different MTU walk orders.
Limitations and Future Work
- Redundancy of MTU Models: The authors observe that the three distinct MTU walk orders (source, target, tree) might be somewhat redundant, with the source order model having the most significant impact.
- Impact of Phrasal Channel Models: Despite the theoretical problems, phrasal channel models still have a significant impact on translation quality. The authors hope to replace them with more effective models that avoid the brittleness and sparsity of heavy lexicalization.
- Future Work: The authors plan to explore other approaches for conditioning on context and to develop more effective replacements for phrasal channel models. They also suggest that MTUs can enable systems to use smaller phrases and phrase tables without sacrificing quality, which is particularly beneficial for syntax-based or discontiguous phrase systems.
Summary updated