Additive Neural Networks for Statistical Machine Translation

Lemao Liu, Taro Watanabe, Eiichiro Sumita, Tiejun ZhaoACL 2013 · OpenReview

Abstract

Most statistical machine translation (SMT) systems are modeled using a log-linear framework. Although the log-linear model achieves success in SMT, it still suffers from some limitations: (1) the features are required to be linear with respect to the model itself; (2) features cannot be further interpreted to reach their potential. A neural network is a reasonable method to address these pitfalls. However, modeling SMT with a neural network is not trivial, especially when taking the decoding efficiency into consideration. In this paper, we propose a variant of a neural network, i.e. additive neural networks, for SMT to go beyond the log-linear translation model. In addition, word embedding is employed as the input to the neural network, which encodes each word as a feature vector. Our model outperforms the log-linear translation models with/without embedding features on Chinese-to-English and Japanese-to-English translation tasks.

Introduction

Presented at ACL 2013, this work represents a pivotal moment in the transition from traditional Statistical Machine Translation (SMT) to the neural era. It introduces Additive Neural Networks (AdNN), a hybrid architecture that allows researchers to reap the representational benefits of neural networks without losing the decoding efficiency of classical search-based translation.

The Problem: Linearity and Shallow Representations

In 2013, the state-of-the-art in SMT was the log-linear model. While powerful, it faced two major flaws:

  1. Linearity Constraint: It assumed that the relationship between features and translation quality was linear. Non-linear interactions had to be manually engineered.
  2. Shallow Interpretation: The model could not "learn" complex representations of words; it simply assigned weights to surface-level counts.

While Neural Networks (NNs) could solve these issues, they were difficult to integrate into SMT because the "decoding" process—a heuristic search through millions of possible translations—is computationally expensive. Calculating a full neural network score for every state expansion would break the efficiency of standard search algorithms like cube pruning.

The Additive Neural Network (AdNN) Approach

The authors introduce the AdNN, which splits the translation score into two parts:

  • The Linear Component: Handles "non-local" features like the Language Model (LM). These are state-dependent and are handled using standard SMT techniques.
  • The Non-Linear (Neural) Component: Encodes "local" features—specifically, the translation rules themselves. Because these rules are fixed once the model is loaded, their neural scores can be pre-computed or cached, preserving the speed of the global search.

Key innovations include using word embeddings as input and a joint training pipeline that learns neural network weights and global feature weights simultaneously using a max-margin objective.

Experimental Results

The model was evaluated on Chinese-to-English and Japanese-to-English tasks. On the Chinese-to-English task, AdNN-Hiero-E achieved a BLEU score of 19.42 on NIST08, a significant jump over the log-linear baseline.

BLEU scores for Chinese-to-English translation tasks across different test sets.

A critical insight from the results is that word embeddings are only useful if the model has the non-linear "depth" (provided by the NN layer) to interpret them. The "L-Hiero-E" model, which added embeddings to a standard linear model, performed poorly compared to the AdNN.

Furthermore, the additive approach successfully maintained decoding efficiency. Because the neural scores for rules do not change during the search process, the decoding time for AdNN is nearly identical to the baseline.

Decoding efficiency comparison between the baseline L-Hiero and the proposed AdNN-Hiero-E.

Conclusion

Additive Neural Networks provide a robust framework for integrating deep learning into SMT. By restricting non-linearity to local rules, the model gains the ability to use continuous word representations without sacrificing the algorithmic efficiency required for large-scale translation tasks. Future work may explore deeper networks and broader bilingual context to further improve performance.

Summary updated

Abstract

Most statistical machine translation (SMT) systems are modeled using a loglinear framework. Although the log-linear model achieves success in SMT, it still suffers from some limitations: (1) the features are required to be linear with respect to the model itself; (2) features cannot be further interpreted to reach their potential. A neural network is a reasonable method to address these pitfalls. However, modeling SMT with a neural network is not trivial, especially when taking the decoding efficiency into consideration. In this paper, we propose a variant of a neural network, i.e. additive neural networks, for SMT to go beyond the log-linear translation model. In addition, word embedding is employed as the input to the neural network, which encodes each word as a feature vector. Our model outperforms the log-linear translation models with/without embedding features on Chinese-to-English and Japanese-to-English translation tasks.
BibTeX
@inproceedings{liu2013additive,
  author = {Lemao Liu and Taro Watanabe and Eiichiro Sumita and Tiejun Zhao},
  title = {{Additive Neural Networks for Statistical Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2013},
  url = {https://openreview.net/forum?id=HkbCg6l_br},
}

Loading related content...

Figure