Abstract
This paper describes a novel target-side syntactic language model for phrase-based statistical machine translation, the bilingual structured language model (BiSLM). Our approach represents a new way to adapt structured language models to statistical machine translation, and a first attempt to adapt them to phrase-based statistical machine translation. We propose a number of variations of the bilingual structured language model and evaluate them in a series of rescoring experiments. Rescoring of 1000-best translation lists produces statistically significant improvements of up to 0.7 BLEU over a strong baseline for Chinese-English, but does not yield improvements for Arabic-English.
Introduction
Standard Statistical Machine Translation (SMT) systems, particularly Phrase-Based SMT (PBSMT), typically rely on $n$-gram language models. While efficient, $n$-grams are limited to local word windows (usually 3–5 words), failing to capture underlying grammatical structures or long-distance dependencies, such as a subject and verb separated by a long prepositional phrase.
Integrating target-side syntax into PBSMT is challenging because decoders build sentences incrementally, lacking the full context required for traditional parsing. Previous attempts often struggled because parsers are designed to find some structure for any input, rather than penalizing ungrammatical translation hypotheses. This paper introduces Bilingual Structured Language Models (BiSLMs) to address these limitations by projecting source-side syntax onto the target side.
Methodology: The BiSLM Approach
The BiSLM adapts the Structured Language Model (SLM)—originally for monolingual speech recognition—to a bilingual SMT setting. The core innovation lies in bypassing on-the-fly target-side parsing during decoding.
Structural Context and Exposed Heads
Instead of conditioning a word only on its linear predecessors, the SLM conditions it on "exposed heads"—the roots of partial syntactic trees built during the incremental generation of the sentence. This allows the model to capture long-range dependencies that linear models miss.
Dependency Projection
The BiSLM uses a "shortcut" based on the Direct Correspondence Assumption (DCA), which assumes that translated equivalents share syntactic relationships. The process follows three main steps:
- Source Parsing: The source sentence is parsed once using a standard dependency parser before translation begins.
- Projection: Using word alignments, the source-side dependency structure is projected onto the target-side hypothesis.
- Sequential Scoring: As the translation is generated, the model identifies exposed heads and conditions the next word on these structural heads.
To handle the complexities of real-world translation, such as many-to-one alignments or unaligned words, the authors define deterministic projection rules (R1–R4).
Experimental Evaluation
The models were evaluated on two major tasks:
- Chinese-English: Trained on ~2.1M sentence pairs. Evaluated on NIST MT06 and MT08.
- Arabic-English: Trained on ~4.3M sentence pairs. Evaluated on NIST MT08 and MT09.
- Baseline: A 5-gram language model trained on the 1.6-billion-token English Gigaword corpus.
Key Results
- Chinese-English: The BiSLM achieved significant improvements of up to +0.7 BLEU over a strong PBSMT baseline. The best results came from variants that did not force unaligned target words to attach to the preceding head, maintaining a cleaner partial parse.
- Arabic-English: The model did not yield improvements. The authors attribute this to the fact that Arabic-English reordering is often more local and better captured by standard $n$-gram models.
Discussion and Limitations
The BiSLM's effectiveness is language-dependent, excelling where significant structural reordering occurs (like Chinese-English). However, the model currently relies on the DCA, which is a simplification that ignores syntactic divergence between languages. Furthermore, as a rescoring model, its impact is limited to the hypotheses already generated by the baseline decoder; full integration into the first-pass decoding process remains a promising area for future work.
Conclusion
Bilingual Structured Language Models provide a computationally feasible way to incorporate target-side syntax into phrase-based SMT. By leveraging source-side parsing and dependency projection, the BiSLM captures long-distance relationships that $n$-gram models miss, leading to measurable improvements in translation quality for structurally divergent language pairs.
Summary updated