Incorporating Syntactic Uncertainty in Neural Machine Translation with Forest-to-Sequence Model

Poorya ZareMoodi, Gholamreza Haffari

TL;DR

This paper introduces a novel Neural Machine Translation (NMT) model that leverages "forests" of parse trees instead of a single parse tree to better handle syntactic uncertainty. By considering multiple possible syntactic structures for a source sentence, their "forest-to-sequence" model significantly improves translation accuracy, especially for languages with complex syntax or when parser errors are present.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) models, while powerful, often struggle with low-resource language pairs due to their heavy reliance on massive amounts of parallel training data. Incorporating syntactic information from the source language has shown promise in improving NMT performance, particularly for linguistically distant languages. However, existing methods that use syntactic information typically rely on a single "best" parse tree generated by a parser. This approach is problematic because parsers are not perfect and can introduce errors, which then negatively impact the translation quality. The paper aims to address this limitation by finding a way to robustly incorporate syntactic information despite potential parsing errors.

Methods

The core innovation of this paper is the "forest-to-sequence" model. Instead of using a single, best parse tree, it utilizes a "packed forest" which represents a collection of exponentially many possible parse trees for a source sentence. This forest is generated by a parser that can output multiple parse hypotheses.

The model then employs a novel encoder that processes this forest. Phrase embeddings are computed using a bottom-up, dynamic programming approach that considers all the trees within the forest. This allows the model to capture a richer representation of the source sentence's syntactic structure, accounting for various interpretations. The decoder then uses an attention mechanism that can attend to both words and these phrase embeddings derived from the forest. This contrasts with previous "tree-to-sequence" models that only attend to words and phrase embeddings from a single tree.

Datasets

The experiments were conducted on three language pairs:

  • English (En) → Chinese (Ch): Using the BTEC corpus.
  • English (En) → Persian (Fa): Using the TEP dataset.
  • English (En) → German (De): Using the first 100K sentences from the Europarl v7 corpus.

Standard preprocessing steps were applied, including lowercasing, tokenization (using the Moses toolkit), and filtering of sentences longer than 50 words. Words with a frequency less than 5 were replaced with an <unk> token. Compact forests and trees were obtained using a parser (Huang, 2008).

Key Insights and Ideas

  • Syntactic Uncertainty is Key: The paper strongly suggests that explicitly modeling the uncertainty in syntactic parsing, rather than relying on a single "best" parse, is crucial for improving NMT robustness.
  • Forests as a Richer Representation: A forest of parse trees provides a more comprehensive view of a sentence's potential syntactic structures than a single tree, allowing the model to compensate for parser errors.
  • Dynamic Programming for Efficiency: The use of dynamic programming to compute phrase embeddings from the forest is essential for managing the computational complexity that would arise from processing each tree individually.
  • Benefits Across Sentence Lengths: Contrary to the intuition that forests would only benefit longer, more complex sentences, the results show improvements across all sentence lengths, indicating the general utility of this approach.
  • Higher Attention to Phrases: The forest-based model shows a tangibly higher ratio of attention on phrase embeddings compared to word embeddings, suggesting that the model effectively utilizes the richer syntactic information provided by the forest.

Key Results

The proposed "forest-to-sequence" model consistently outperformed both the vanilla sequence-to-sequence NMT model and the "tree-to-sequence" model across all three language pairs (En→De, En→Ch, En→Fa). For instance, on the English→Persian dataset, the forest-to-sequence model achieved a higher BLEU score (11.91 with a hidden dimension of 256) compared to the tree-to-sequence model (11.71) and the vanilla ANMT model (10.93). The paper also shows that the forest model is beneficial regardless of sentence length, and the highest improvements were observed on the English→Persian dataset, which is noted as being the "noisiest" and having the highest parsing uncertainty.

Limitations and Future Work

The paper acknowledges that the performance of their model is still dependent on the quality of the parser that generates the forests. While their method mitigates the impact of single-tree errors, a very poor parser might still limit the benefits. The authors suggest that future work could explore more advanced methods for combining phrase embeddings from the forest and potentially investigate different parsing techniques or forest representations. The paper also notes that the generated trees were not necessarily binary, which is a detail handled by their forest-LSTM encoder.

Summary updated

Abstract

Incorporating syntactic information in Neural Machine Translation models is a method to compensate their requirement for a large amount of parallel training text, especially for low-resource language pairs. Previous works on using syntactic information provided by (inevitably error-prone) parsers has been promising. In this paper, we propose a forest-to-sequence Attentional Neural Machine Translation model to make use of exponentially many parse trees of the source sentence to compensate for the parser errors. Our method represents the collection of parse trees as a packed forest, and learns a neural attentional transduction model from the forest to the target sentence. Experiments on English to German, Chinese and Persian translation show the superiority of our method over the tree-to-sequence and vanilla sequence-to-sequence neural translation models.
BibTeX
@inproceedings{zaremoodi2018incorporating,
  author = {Poorya ZareMoodi and Gholamreza Haffari},
  title = {{Incorporating Syntactic Uncertainty in Neural Machine Translation with Forest-to-Sequence Model}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=IR95S8rtI6},
}
Figure