Abstract
Data-Oriented Translation (DOT) models traditionally rely on the statistical likelihood of tree fragments to select translations. However, likelihood often fails to correlate with translation accuracy. This paper introduces an accuracy-based rescoring framework for DOT that weights tree fragments based on their similarity to "oracle" translations. Using tree-edit distance for structured rescoring, the authors achieve a significant relative improvement of up to 20.6% in BLEU scores on Spanish-to-English translation tasks.
Introduction
Data-Oriented Translation (DOT) is a syntax-based approach to Machine Translation that builds new translations from linked tree fragments extracted from a parallel corpus. Traditionally, these models select the optimal translation by maximizing the likelihood of the derivation—essentially favoring the most frequent fragments.
A fundamental flaw in this approach is that likelihood and accuracy do not necessarily correlate. A fragment may be statistically frequent in training data but lead to poor translations in novel contexts. The challenge addressed in this research is to "point the decoder" toward translations that resemble high-quality human output rather than merely the most statistically probable ones.
Methods
The authors reformulate DOT scoring into a log-linear framework, allowing the combination of traditional likelihood with new accuracy-based features. To manage the exponential number of possible fragments, they utilize the Goodman reduction, which transforms the DOT model into an isomorphic Probabilistic Context-Free Grammar (PCFG). This transformation makes the model computationally efficient while preserving its structural properties.
The core of the proposal lies in three rescoring methods:
- Structured Fragment Rescoring (SFR): This method uses a recursive tree-edit distance to compare a candidate fragment against an "oracle" fragment (the best possible fragment for a specific source span). It accounts for structural changes such as inversions, insertions, and deletions.
- Normalized SFR (NSFR): This variation normalizes the edit distance by fragment size to prevent bias against larger, more complex structures.
- Fragment Surface Rescoring (FSR): A simpler approach that ignores internal tree structure, using Damerau-Levenshtein distance to compare the actual word sequences (surface yields) of candidate and oracle fragments.
Experimental Setup
Experiments were conducted using the Spanish-to-English portion of the Europarl corpus. The training set consisted of 10,000 sentence pairs, with up to 40,000 pairs used for tuning/rescoring. Evaluation was performed on 2,000 randomly selected sentences. Oracle fragments were identified using the GTM (General Text Matcher) F-score, which provides a more reliable sentence-level evaluation than BLEU.
Results and Discussion
All three rescoring methods significantly outperformed the relative-frequency baseline. The best-performing system, SFR, reached a BLEU score of 12.12 on the 40K tuning set, compared to the baseline of 10.82—a 20.6% relative improvement.
Key insights from the results include:
- Structure Matters: SFR consistently outperformed the surface-level FSR, suggesting that the syntactic structure in DOT fragments provides a valuable signal for translation quality that words alone cannot capture.
- Avoiding the "Likelihood Trap": Qualitative analysis showed that the baseline likelihood model often decomposed phrases into too many small, frequent sub-derivations, leading to nonsensical outputs. The accuracy-based model encouraged fragments that preserved proper word order and idiomatic phrasing.
Limitations and Future Work
While promising, the study was limited by its scale (10k-40k sentences) compared to modern systems using millions of pairs. Additionally, the lack of an external language model and the manual setting of log-linear weights suggest that further improvements are possible. Future work will explore automated tuning methods like Minimum Error Rate Training (MERT) and the integration of language models to enhance absolute translation quality.
Summary updated