Abstract
This paper addresses the problem of producing a diverse set of plausible translations. We present a simple procedure that can be used with any statistical machine translation (MT) system. We explore three ways of using diverse translations: (1) system combination, (2) discriminative reranking with rich features, and (3) a novel post-editing scenario in which multiple translations are presented to users. We find that diversity can improve performance on these tasks, especially for sentences that are difficult for MT.
Introduction
Standard Machine Translation (MT) systems typically generate $M$-best lists where candidates are highly redundant, often differing only by minor punctuation or suffixes. This redundancy stems from the structured search space where top-scoring paths share most edges. This paper introduces "dissimilarity-augmented decoding" to inject diversity into these lists, arguing that a set of diverse, plausible translations provides better "raw material" for downstream tasks like system combination or human post-editing.
Methodology: Dissimilarity-Augmented Decoding
The authors propose a greedy, iterative procedure to generate diverse lists. Starting with the model-best translation ($y^1$), each subsequent translation $y^m$ is found by optimizing a balance between the original model score and a dissimilarity term:
$$\text{argmax}_{y} [ \text{ModelScore}(y) + \sum_{j=1}^{m-1} \lambda_j \Delta(y^j, y) ]$$
Where $\Delta$ is a function measuring the difference between a candidate and previously selected translations. Remarkably, this dissimilarity term can be implemented as an additional "diversity" language model in standard ARPA format, allowing the approach to be used with existing MT decoders (like Moses) without modifying underlying search algorithms.
Experimental Results
Oracle BLEU and Efficiency
The potential quality of diverse lists is significantly higher than standard $M$-best lists. As shown in the table below, a list of just 20 diverse translations achieves Oracle BLEU scores comparable to a standard 1000-best list while being far more efficient. For Chinese-to-English, 20 diverse candidates actually match the oracle performance of the full 1000-best list.
System Combination and the Difficulty Threshold
Diversity is particularly effective for "difficult" sentences where the baseline system struggles. In system combination tasks, using diverse lists led to gains of 1.2 to 2.5 BLEU points for the bottom quartile of performance (q1). Conversely, for high-confidence sentences (q4), the baseline often remains superior, suggesting that forcing diversity can introduce noise when the system is already highly accurate.
Discriminative Reranking
When training a discriminative reranker, diverse lists act as a regularizer, preventing the model from overfitting to a narrow probability region. Combining diverse lists with rich features (e.g., syntactic LMs, word clusters, and Google 5-grams) yielded a 1.7 BLEU point gain over the baseline for Arabic-English. Using diverse lists allowed a reranker to achieve the performance of a 200-best list using only 20 diverse candidates.
Conclusion
Dissimilarity-augmented decoding offers a simple yet powerful way to improve MT utility. By forcing the explorer to look beyond the immediate vicinity of the best path, we gain significant improvements in downstream performance and human interpretability, especially in challenging translation scenarios. Future work will look into integrating diversity directly into search algorithms and exploring stylistic or syntactic diversity.
Summary updated