Introduction
Statistical Machine Translation (SMT), particularly hierarchical models (Hiero), relies on a set of features to score translation hypotheses. Traditionally, these features are manually engineered by experts to capture linguistic properties like syntax or word reordering. However, manual engineering is labor-intensive and may fail to capture complex, non-linear relationships between languages. This paper proposes an unsupervised method to automatically induce "deep features" using Deep Belief Networks (DBNs) to enhance translation performance.
Methodology
The proposed approach integrates DBNs into the Hiero framework. The process begins with the four standard baseline features used in Hiero translation rules. These features serve as the visible nodes for a DBN composed of stacked Restricted Boltzmann Machines (RBMs).
The network is trained using Contrastive Divergence in a completely unsupervised manner, requiring only the existing rule tables rather than human-labeled data. Once trained, the baseline features of each rule are passed through the DBN. The activations from the hidden layers are extracted and appended to the original rule as new, high-dimensional "deep features." This allows the model to leverage non-linear transformations and latent correlations between the basic features that the baseline log-linear model cannot capture.
Experimental Results
The method was evaluated on Dari-English and Chinese-English speech translation tasks. The baseline performance for the Dari-English task (represented as English-to-Farsi/F2E) is summarized in the table below.
Impact of Network Architecture and Depth
A key finding of the research is that the topology and depth of the DBN significantly influence translation quality. The authors explored various architectures, including parallel, pyramid, and inverse pyramid shapes.
The results indicate a "sweet spot" for network depth. While increasing the number of layers initially improves the BLEU score by providing more discriminative features, excessive depth eventually leads to diminishing returns or performance degradation due to overtraining.
As shown in the experimental data, the 4x4x4x4 architecture provided a substantial gain over the baseline, reaching a test BLEU score of 0.1522 for the Farsi-to-English (F2E) task, compared to the baseline of 0.1422. This represents a significant improvement achieved without manual linguistic engineering.
Conclusion
This work demonstrates that unsupervised deep learning can effectively refine existing SMT pipelines. By automatically inducing features that capture complex dependencies, the DBN-based approach achieved significant BLEU score improvements (+1.13 for Dari-to-English) without requiring additional human annotation. Future research will focus on automatically inducing the optimal network structure based on data size and language pairs to further streamline the feature induction process.
Summary updated