Transformer-Based Direct Hidden Markov Model for Machine Translation

Weiyue Wang, Zijian Yang, Yingbo Gao, Hermann NeyACL 2021

TL;DR

This paper introduces a novel approach to machine translation by integrating the principles of Hidden Markov Models (HMMs) directly into the Transformer architecture. This hybrid model achieves competitive translation quality, particularly outperforming the standard Transformer in terms of Translation Edit Rate (TER), while offering a more interpretable alignment mechanism.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) systems, especially those based on the Transformer architecture, have achieved state-of-the-art performance. A key component in these systems is the attention mechanism, which implicitly learns alignments between source and target sentences. However, the attention mechanism's alignment quality can be suboptimal, and it lacks the explicit probabilistic framework of traditional alignment models like Hidden Markov Models (HMMs). While previous work has explored HMMs with recurrent neural networks (RNNs), these approaches have been surpassed by Transformers. This paper aims to bridge this gap by incorporating HMM principles directly into the Transformer architecture, seeking to improve translation performance and alignment interpretability without sacrificing the Transformer's monolithic structure.

Methods

The core innovation is the integration of a direct HMM formulation within the Transformer's decoder layers. Instead of relying solely on the attention mechanism for alignment, the proposed model explicitly models both lexicon probabilities (the probability of a target word given a source word) and alignment probabilities (the probability of a target word aligning to a specific source word).

  • Direct HMM Formulation: The paper adapts the HMM concept to model $P(e|f)$, the probability of a target sentence $e$ given a source sentence $f$, directly. This is achieved by introducing an alignment variable $b_i$ representing the source position aligned to the $i$-th target word.
  • Transformer Integration: The HMM components are integrated into the Transformer's decoder. Specifically, the cross-attention mechanism is modified to incorporate alignment information.
  • Zero-Order vs. First-Order HMM:
    • Zero-Order: This simpler model assumes the alignment of the current target word depends only on the current source word and the previously generated target words. It modifies the attention mechanism by moving the weighted sum outside the ReLU activation and projecting the target hidden state before adding it to the source hidden state.
  • First-Order: This model introduces a dependency on the previous alignment position ($b_{i-1}$), making the alignment probability $P(b_i | b_{i-1}, e^{<i}, f)$. This increases computational complexity, requiring a dynamic programming approach (forward path) for training.
  • Training: The paper leverages automatic differentiation for training, which is more efficient than manual differentiation, especially when label smoothing is applied to the lexicon probability.

Datasets

The experiments were conducted on standard machine translation benchmarks:

  • WMT 2019 German-English (de-en)
  • WMT 2019 Chinese-English (zh-en)
  • WMT 2018 English-Turkish (en-tr)

These datasets vary in size, from hundreds of thousands to tens of millions of sentence pairs. The paper also mentions using the RWTH German-English Golden Alignments corpus for evaluating alignment quality.

Key Insights and Ideas

  • HMMs within Transformers are Viable: The paper demonstrates that directly integrating HMM principles into the Transformer architecture is feasible and can lead to improved performance.
  • Zero-Order HMM is Surprisingly Effective: The zero-order direct HMM model achieves competitive translation quality and outperforms the baseline Transformer in TER scores, even without the complexity of first-order dependencies. This suggests that a significant portion of the alignment information can be captured without explicit sequential dependencies.
  • First-Order Dependency Offers Limited Gains: Despite the theoretical appeal of first-order dependencies, experimental results show they do not provide further improvements over the zero-order model. Heatmaps suggest the model tends to "forget" this explicit dependency during training, likely due to the complexity and the model's tendency to find simpler, more focused alignments.
  • Improved Alignment Quality: The direct HMM, particularly when considering the posterior probability, provides better alignment quality (lower AER) compared to standard Transformer attention mechanisms. This is a significant advantage as it offers more interpretable alignments.
  • Monolithic System: The proposed approach keeps the system monolithic, integrating alignment modeling directly into the Transformer without requiring external alignment tools or separate language models for supervision.

Key Results

  • Translation Performance: The direct HMM models achieved comparable BLEU scores to the baseline Transformer but outperformed it in TER scores across multiple language pairs. For example, the zero-order HMM achieved a TER of 47.7% on German-English, compared to 48.2% for the Transformer baseline. The first-order HMM also showed slight improvements in TER.
  • Alignment Quality: The direct HMM's alignment probability output resulted in a significantly lower Alignment Error Rate (AER) of 31.8% on the German-English alignment task, compared to 39.1% for Transformer attention from the fifth decoder layer and 55.7% from the sixth. The posterior probability further improved this to 27.4%.
  • Efficiency: The zero-order HMM has a slight disadvantage in training speed due to GPU memory constraints (requiring smaller batch sizes) but is only slightly slower in decoding than the Transformer baseline. The first-order HMM is significantly slower in both training and decoding due to its increased complexity.

Limitations and Future Work

  • First-Order Dependency Underutilization: The paper acknowledges that the first-order dependency is not effectively learned by the current model. Future work could explore constraints or other techniques to better leverage this information.
  • Alignment Quality vs. Translation Performance: While the primary goal was to improve translation performance, the paper notes that the alignment quality, though improved, still lags behind specialized alignment tools like GIZA++. Future work could focus on further enhancing alignment quality.
  • Combining Attention and Alignment: The authors suggest that combining the attention mechanism with the explicit alignment information from the HMM could lead to further performance improvements.
  • Decoding Speed: While the zero-order HMM is only slightly slower, the first-order HMM's decoding speed is a significant bottleneck. Further research into efficient decoding for first-order HMMs within Transformers would be beneficial.

Summary updated

BibTeX
@inproceedings{wang2021transformer,
  author = {Weiyue Wang and Zijian Yang and Yingbo Gao and Hermann Ney},
  title = {{Transformer-Based Direct Hidden Markov Model for Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure