Generating Diverse Translation from Model Distribution with Dropout

Xuanfu Wu, Yang Feng, Chenze Shao

Abstract

Standard Neural Machine Translation (NMT) models typically optimize for a single "best" output, often resulting in repetitive and overly cautious translations that lack the natural variety of human language. This paper presents a Bayesian approach to NMT that treats model parameters as a posterior distribution rather than fixed values. By employing "concrete dropout"—a continuous relaxation of the dropout operation—as a trainable parameter, we enable the model to represent inherent linguistic uncertainty. During inference, sampling from this learned distribution allows for the generation of diverse, high-quality translations that maintain grammatical and semantic integrity.

Introduction

While NMT has achieved significant milestones in translation accuracy, the problem of diversity remains a challenge. Natural language is inherently flexible, with multiple valid ways to express the same concept through different lexical and syntactic choices. However, standard decoding strategies like beam search tend to produce candidates that are nearly identical, differing only by minor word substitutions. Previous attempts to address this, such as adding diversity penalties to beam search or using complex latent variable models, often fail to capture the model's internal uncertainty. This work leverages Bayesian Neural Networks (BNNs) to directly model this uncertainty, providing a principled framework for diverse generation.

Methodology

Bayesian Neural Networks and Variational Inference

The core of our approach is treating the NMT model parameters not as static weights, but as a posterior distribution conditioned on the training data. Because calculating the exact posterior is computationally intractable, we utilize Variational Inference (VI) to approximate it. We optimize a variational distribution to minimize the Kullback-Leibler (KL) divergence from the true posterior.

Concrete Dropout

To make the dropout probability ($p$) a learnable parameter, we use "concrete dropout." Unlike standard dropout, which uses a fixed rate, concrete dropout allows the model to learn which specific modules or layers can afford higher variance. We fine-tune a pre-trained Transformer model by optimizing these dropout probabilities alongside the standard translation loss. This process identifies "flexible" parts of the network where diversity can be introduced without significantly degrading accuracy.

Inference via Sampling

At inference time, we generate diverse translations by performing multiple forward passes. In each pass, a different "expert" model is effectively sampled from the learned distribution by masking weights according to the trained dropout probabilities. This allows the system to explore the manifold of high-probability translations learned during training.

Experimental Setup

We evaluated the proposed method on two major benchmarks:

  • NIST Chinese-to-English (Zh-En): ~1.34 million sentence pairs.
  • WMT’14 English-to-German (En-De): ~4.5 million sentence pairs.

Performance was measured using BLEU for accuracy and Pairwise-BLEU for diversity. A lower Pairwise-BLEU score indicates higher diversity among the generated candidates.

Results and Discussion

Accuracy-Diversity Trade-off

The proposed method was compared against several baselines, including Diverse Beam Search (DBS), HardMoE, and Head Sampling. Our results demonstrate a superior Pareto front across both datasets. As shown in the Zh-En evaluation, the "Full Model" approach consistently achieves a better balance, providing higher diversity (lower Pairwise-BLEU) for a given level of accuracy compared to traditional diversity-promoting heuristics.

Accuracy vs. Diversity Pareto front on the NIST Zh-En dataset.

Qualitative Analysis

Qualitative results highlight the difference between heuristic-based diversity and our Bayesian approach. While beam search often produces variations that only swap single synonyms, our method generates translations with distinct syntactic structures. For example, it can shift from "An important item on the agenda..." to "One of the important topics for discussion...", reflecting a more human-like grasp of paraphrasing.

Comparison of translation diversity between beam search and the proposed Bayesian dropout method.

Module Importance and Correlation

A key finding of this study is the strong correlation between learned dropout probabilities and module importance. We observed a correlation of up to $\rho = 0.919$ between the learned dropout rate and the drop in BLEU score when a module is pruned. This suggests that the Bayesian framework effectively identifies redundant or flexible components, concentrating diversity in areas where the model is least certain.

Conclusion

By treating NMT models as distributions via concrete dropout, we provide a robust method for generating diverse translations. This approach stays within the learned manifold of the data, ensuring that diversity does not come at the cost of basic grammatical correctness. While sampling-based inference is computationally more intensive than a single beam search, the resulting gains in linguistic variety offer a significant step forward for applications requiring creative or varied language generation. Future work will focus on optimizing sampling efficiency and extending this framework to tasks like summarization and dialogue systems.

Summary updated

BibTeX
@inproceedings{wu2020generating,
  author = {Xuanfu Wu and Yang Feng and Chenze Shao},
  title = {{Generating Diverse Translation from Model Distribution with Dropout}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=_1DclMQh1m5},
}
Figure