Unsupervised Deep Belief Features for Speech Translation

Sameer Maskey, Bowen ZhouInterspeech 2012 · DOI

Introduction

Statistical Machine Translation (SMT), particularly hierarchical models (Hiero), relies on a set of features to score translation hypotheses. Traditionally, these features are manually engineered by experts to capture linguistic properties like syntax or word reordering. However, manual engineering is labor-intensive and may fail to capture complex, non-linear relationships between languages. This paper proposes an unsupervised method to automatically induce "deep features" using Deep Belief Networks (DBNs) to enhance translation performance.

Methodology

The proposed approach integrates DBNs into the Hiero framework. The process begins with the four standard baseline features used in Hiero translation rules. These features serve as the visible nodes for a DBN composed of stacked Restricted Boltzmann Machines (RBMs).

The network is trained using Contrastive Divergence in a completely unsupervised manner, requiring only the existing rule tables rather than human-labeled data. Once trained, the baseline features of each rule are passed through the DBN. The activations from the hidden layers are extracted and appended to the original rule as new, high-dimensional "deep features." This allows the model to leverage non-linear transformations and latent correlations between the basic features that the baseline log-linear model cannot capture.

Experimental Results

The method was evaluated on Dari-English and Chinese-English speech translation tasks. The baseline performance for the Dari-English task (represented as English-to-Farsi/F2E) is summarized in the table below.

Baseline BLEU scores for English-to-Farsi (E2F) and Farsi-to-English (F2E) tasks.

Impact of Network Architecture and Depth

A key finding of the research is that the topology and depth of the DBN significantly influence translation quality. The authors explored various architectures, including parallel, pyramid, and inverse pyramid shapes.

The results indicate a "sweet spot" for network depth. While increasing the number of layers initially improves the BLEU score by providing more discriminative features, excessive depth eventually leads to diminishing returns or performance degradation due to overtraining.

BLEU scores across different Deep Model architectures, illustrating the impact of increasing network depth on translation performance.

As shown in the experimental data, the 4x4x4x4 architecture provided a substantial gain over the baseline, reaching a test BLEU score of 0.1522 for the Farsi-to-English (F2E) task, compared to the baseline of 0.1422. This represents a significant improvement achieved without manual linguistic engineering.

Conclusion

This work demonstrates that unsupervised deep learning can effectively refine existing SMT pipelines. By automatically inducing features that capture complex dependencies, the DBN-based approach achieved significant BLEU score improvements (+1.13 for Dari-to-English) without requiring additional human annotation. Future research will focus on automatically inducing the optimal network structure based on data size and language pairs to further streamline the feature induction process.

Summary updated

BibTeX
@inproceedings{maskey2012unsupervised,
  author = {Sameer Maskey and Bowen Zhou},
  title = {{Unsupervised Deep Belief Features for Speech Translation}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-618},
}
Figure