Abstract
This paper introduces four lightweight, probabilistic features designed to enhance hierarchical statistical machine translation (SMT) without the scalability constraints of traditional sparse lexical features. By integrating non-terminal labels, length distributions, and source-side context, the researchers achieved significant BLEU score improvements—up to 2.0 points—in Arabic-to-English and Chinese-to-English translation tasks.
Introduction
In the late 2000s, the potential of "lexical features"—which capture information about specific words and their immediate contexts—to improve machine translation was well-recognized. However, scaling these features presented a significant challenge. Researchers faced a "Goldilocks" problem: discriminative online learning methods like Stochastic Gradient Descent (SGD) allowed for rich feature sets but were prone to over-fitting and required massive development sets for tuning. Conversely, Maximum Entropy (MaxEnt) models avoided over-fitting but struggled with global optimality or required impractical data filtering for real-world applications.
This study seeks a middle ground by introducing features that capture linguistic and contextual nuances while remaining computationally efficient and robust enough for integration into state-of-the-art hierarchical decoders.
Methodology
The authors implemented four feature functions within BBN’s HierDec, a string-to-dependency decoder. These features are designed to be lightweight and probabilistic, avoiding the overhead of massive sparse feature sets.
Non-terminal Labels
In standard hierarchical SMT, a generic "X" is used for all placeholders in a translation rule. The authors instead assign Part-of-Speech (POS) tags based on the head word of the target side. This is implemented as a "soft constraint," where the decoder is penalized if labels do not match during substitution.
Length Distribution
To improve the selection of translation rules, the authors modeled the expected length of a source span for a given non-terminal using a smoothed Gaussian distribution. This helps the system determine if a rule is appropriate for the physical size of the phrase being translated, providing more consistent gains than previous attempts using Poisson or simple count distributions.
Context Language Model (CLM)
This feature utilizes a tri-gram model to examine the source words immediately adjacent to the current span. By looking at the left and right context, the CLM helps disambiguate which translation rule is most appropriate for the specific source text environment.
Source Dependency LM
The decoder was designed to simultaneously generate source and target dependency trees. This feature scores the likelihood of the source structure, intended as a structural "sanity check" for the translation hypothesis.
Results and Analysis
The models were evaluated on substantial datasets, including 29 million Arabic words and 107 million Chinese words. Systems were tuned on NIST MT02-05 and tested on the MT06 and MT08 newswire sets.
The cumulative impact of the proposed features was significant:
- Arabic-to-English: The combination of labels, length, and context features improved lower-cased BLEU by 2.0 points on MT06 and 1.7 points on MT08.
- Chinese-to-English: The same configuration yielded improvements of 1.0 BLEU on MT06 and 0.8 on MT08.
Interestingly, the Source Dependency LM provided no improvement. The authors attribute this to structural mismatches between source and target trees generated by independent parsers, which made it difficult for the model to find meaningful correlations.
Conclusion
This research demonstrates that simple, robust probabilistic models—such as Gaussian length distributions and n-gram context models—can outperform or match more complex MaxEnt approaches in large-scale MT tasks. While structural mismatches limited the effectiveness of source dependency features, the overall approach provides a scalable path for incorporating linguistic and contextual information into hierarchical SMT systems. Future work will focus on unified Gaussian models and improving bilingual parser compatibility to resolve structural discrepancies.
Summary updated