Abstract
Despite the improvement of translation quality, neural machine translation (NMT) often suffers from the lack of diversity in its generation. In this paper, we propose to generate diverse translations by deriving a large number of possible models with Bayesian modelling and sampling models from them for inference. The possible models are obtained by applying concrete dropout to the NMT model and each of them has specific confidence for its prediction, which corresponds to a posterior model distribution under specific training data in the principle of Bayesian modeling. With variational inference, the posterior model distribution can be approximated with a variational distribution, from which the final models for inference are sampled. We conducted experiments on Chinese-English and English-German translation tasks and the results shows that our method makes a better trade-off between diversity and accuracy.
Introduction
Standard neural machine translation (NMT) systems are typically optimized to find a single "best" translation for a given source sentence. However, human language is inherently flexible, with multiple valid ways to express the same concept through varying vocabulary and syntax. While standard decoding techniques like beam search can produce multiple candidates, these often lack meaningful diversity, frequently differing by only minor punctuation or single-word substitutions.
This paper addresses the diversity bottleneck by treating NMT as a Bayesian Neural Network (BNN). Instead of a fixed set of weights, we learn a distribution over possible models. By sampling different "sub-models" from this distribution at inference time, we can generate a wide array of high-quality, diverse translations.
Methodology: Bayesian NMT via Concrete Dropout
The core of our approach is to approximate a posterior distribution of models given the training data. Since exact Bayesian inference is intractable for large-scale NMT, we utilize Concrete Dropout and variational inference.
Concrete Dropout and Variational Inference
Unlike standard dropout, where the dropout rate is a fixed hyperparameter, concrete dropout allows the dropout probabilities to be learned during training via backpropagation. We maximize the Evidence Lower Bound (ELBO), which balances the model's fit to the training data with a regularization term that keeps the learned distribution close to a prior.
Architectural Integration
We apply this learnable dropout to three critical components of the Transformer architecture:
- Embedding Layers: To introduce uncertainty in word representation.
- Multi-Head Attention: To vary how the model focuses on different parts of the source and target sequences.
- Feed-Forward Networks (FFN): To diversify the intermediate transformations of hidden states.
During inference, we perform multiple forward passes. In each pass, a different model mask is sampled from the trained distribution, effectively querying a unique "version" of the model to produce a distinct translation.
Experimental Results
We evaluated our method on the NIST Chinese-to-English (Zh-En) and WMT’14 English-to-German (En-De) benchmarks.
The Diversity-Accuracy Trade-off
Our method demonstrates a superior Pareto front compared to baselines like Diverse Beam Search (DBS) and Hard Mixture of Experts (HardMoE). By adjusting the prior hyperparameter $l$, we can navigate the trade-off between translation accuracy (BLEU) and diversity (measured by lower Pairwise-BLEU).
Qualitative Analysis
Qualitative results highlight the model's ability to generate structurally and lexically diverse sentences. While standard beam search often produces repetitive variations, our sampling-based approach captures different nuances of the source text.
Conclusion
By reframing NMT as a Bayesian inference problem and leveraging learnable concrete dropout, we provide a robust framework for generating diverse translations. Our approach is computationally efficient and outperforms existing diversity-promoting techniques on major benchmarks. Future work will explore applying this Bayesian framework to other sequence generation tasks and refining the sampling strategies during decoding.
Summary updated