Abstract
Users of machine translation systems may desire to obtain multiple candidates translated in different ways. In this work, we attempt to obtain diverse translations by using sentence codes to condition the sentence generation. We describe two methods to extract the codes, either with or without the help of syntax information. For diverse generation, we sample multiple candidates, each of which conditioned on a unique code. Experiments show that the sampled translations have much higher diversity scores when using reasonable sentence codes, where the translation quality is still on par with the baselines even under strong constraint imposed by the codes. In qualitative analysis, we show that our method is able to generate paraphrase translations with drastically different structures. The proposed approach can be easily adopted to existing translation systems as no modification to the model is required.
Introduction
Standard Neural Machine Translation (NMT) systems are optimized to find the most probable translation, which often results in a lack of variety. When generating multiple candidates via traditional methods like beam search, the outputs are frequently near-duplicates with only minor lexical variations. Previous attempts to increase diversity by encouraging "creative" vocabulary often lead to "hallucinations" or semantic drift. This research addresses the challenge of generating multiple, high-quality translations that are structurally distinct (paraphrases) while remaining faithful to the source text.
Methodology
The proposed approach introduces "sentence codes"—discrete indicators of semantic or syntactic structure—to condition the NMT model. This "mixture-of-experts" framework involves two primary components: code extraction and conditioned generation.
Code Extraction
The researchers explored two distinct paradigms for generating these codes:
- Semantic Coding: Target sentences are mapped to embeddings using pre-trained models (BERT or FastText), which are then clustered via k-means. The resulting cluster ID serves as the code.
- Syntactic Coding: To isolate structural information, a constituency parser extracts the sentence's tree structure (stripping lexical content). A TreeLSTM-based auto-encoder with a "discretization bottleneck" (using improved semantic hashing) then compresses this tree into an 8-bit binary code.
Training and Inference
During training, a standard Transformer model is fed the source sentence and a target sentence prefixed with its corresponding code (e.g., [code_12] [eoc] The cat sat on the mat). At inference, the model predicts the most likely codes for a given source sentence and generates a unique translation for each. Notably, this method requires no architectural modifications to the NMT system.
Experimental Evaluation
The models were evaluated on two major benchmarks:
- ASPEC (Japanese-to-English): 3 million bilingual pairs from formal scientific excerpts.
- WMT14 (German-to-English): 4.5 million bilingual pairs featuring informal news text.
Analysis of Results
The researchers utilized a Diversity Score (DP), based on BLEU-based discrepancy, to quantify the structural variety of generated candidates.
Key Findings
- Syntax-Driven Diversity: Syntactic coding significantly outperformed semantic clustering in diversity gains. Since the source sentence already provides semantic grounding, the code is most effective when it dictates structural arrangement (syntax).
- Structural Paraphrasing: Qualitative analysis revealed that the model learned to associate specific codes with distinct grammatical constructions, such as active versus passive voice.
- Quality Preservation: Unlike many diversity-promoting methods that sacrifice accuracy, this approach maintained BLEU scores on par with or slightly better than the baseline. For instance, on ASPEC, syntactic coding achieved 27.4 BLEU compared to the baseline's 27.1, while nearly doubling the DP score (39.8 vs. 22.4).
- Oracle Potential: When provided with the "correct" code from a reference translation (Oracle), BLEU scores improved significantly (e.g., 30.7 on WMT14), indicating that the primary bottleneck is code prediction rather than translation capability.
Conclusion
This work demonstrates that conditioning NMT on discrete sentence codes is an effective way to generate diverse, high-quality translations. While the syntactic approach is highly effective, it currently depends on high-quality constituency parsers, which may limit its application for low-resource languages. Future work will focus on improving unsupervised semantic code extraction and optimizing the prediction of valid codes for novel source sentences.
Summary updated