Abstract
Most statistical machine translation (SMT) systems are modeled using a log-linear framework. Although the log-linear model achieves success in SMT, it still suffers from some limitations: (1) the features are required to be linear with respect to the model itself; (2) features cannot be further interpreted to reach their potential. A neural network is a reasonable method to address these pitfalls. However, modeling SMT with a neural network is not trivial, especially when taking the decoding efficiency into consideration. In this paper, we propose a variant of a neural network, i.e. additive neural networks, for SMT to go beyond the log-linear translation model. In addition, word embedding is employed as the input to the neural network, which encodes each word as a feature vector. Our model outperforms the log-linear translation models with/without embedding features on Chinese-to-English and Japanese-to-English translation tasks.
Introduction
Presented at ACL 2013, this work represents a pivotal moment in the transition from traditional Statistical Machine Translation (SMT) to the neural era. It introduces Additive Neural Networks (AdNN), a hybrid architecture that allows researchers to reap the representational benefits of neural networks without losing the decoding efficiency of classical search-based translation.
The Problem: Linearity and Shallow Representations
In 2013, the state-of-the-art in SMT was the log-linear model. While powerful, it faced two major flaws:
- Linearity Constraint: It assumed that the relationship between features and translation quality was linear. Non-linear interactions had to be manually engineered.
- Shallow Interpretation: The model could not "learn" complex representations of words; it simply assigned weights to surface-level counts.
While Neural Networks (NNs) could solve these issues, they were difficult to integrate into SMT because the "decoding" process—a heuristic search through millions of possible translations—is computationally expensive. Calculating a full neural network score for every state expansion would break the efficiency of standard search algorithms like cube pruning.
The Additive Neural Network (AdNN) Approach
The authors introduce the AdNN, which splits the translation score into two parts:
- The Linear Component: Handles "non-local" features like the Language Model (LM). These are state-dependent and are handled using standard SMT techniques.
- The Non-Linear (Neural) Component: Encodes "local" features—specifically, the translation rules themselves. Because these rules are fixed once the model is loaded, their neural scores can be pre-computed or cached, preserving the speed of the global search.
Key innovations include using word embeddings as input and a joint training pipeline that learns neural network weights and global feature weights simultaneously using a max-margin objective.
Experimental Results
The model was evaluated on Chinese-to-English and Japanese-to-English tasks. On the Chinese-to-English task, AdNN-Hiero-E achieved a BLEU score of 19.42 on NIST08, a significant jump over the log-linear baseline.
A critical insight from the results is that word embeddings are only useful if the model has the non-linear "depth" (provided by the NN layer) to interpret them. The "L-Hiero-E" model, which added embeddings to a standard linear model, performed poorly compared to the AdNN.
Furthermore, the additive approach successfully maintained decoding efficiency. Because the neural scores for rules do not change during the search process, the decoding time for AdNN is nearly identical to the baseline.
Conclusion
Additive Neural Networks provide a robust framework for integrating deep learning into SMT. By restricting non-linearity to local rules, the model gains the ability to use continuous word representations without sacrificing the algorithmic efficiency required for large-scale translation tasks. Future work may explore deeper networks and broader bilingual context to further improve performance.
Summary updated