GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators

Yuchen Hu, Chen Chen 0075, Chao-Han Huck Yang, Ruizhe Li, Dong Zhang, Zhehuai Chen, Engsiong Chng

Introduction

GenTranslate is a generative paradigm designed to enhance multilingual speech and machine translation by leveraging the reasoning capabilities of Large Language Models (LLMs). Unlike traditional translation systems that rely on a single "best" output, GenTranslate synthesizes a higher-quality result by integrating information from a list of $N$-best candidate hypotheses.

The Information Loss Problem

Modern translation systems for speech-to-text (ST) and text-to-text (MT) typically utilize beam search decoding. While this process generates multiple candidate hypotheses, standard practice involves discarding all but the top-ranked one (top-1 selection). This approach often leads to significant "information loss," as the top-1 hypothesis may contain minor errors while correct semantic or grammatical fragments are preserved in lower-ranked candidates. GenTranslate addresses this by treating the $N$-best list as a prompt for an LLM, which acts as a master editor to generate a superior final translation.

Comparison between typical translation (top-1 selection) and the GenTranslate generative paradigm.

Methodology: A Second-Pass Generative Paradigm

GenTranslate operates as a "second-pass" system that refines the output of existing foundation models. The workflow consists of three primary stages:

  1. Candidate Generation: A foundation model, such as Meta’s SeamlessM4T, processes the source input to produce an $N$-best list of hypotheses.
  2. LLM Integration: These candidates are provided to an LLM alongside a specific instruction to integrate the versions into a higher-quality translation.
  3. Efficient Fine-tuning: To maintain computational efficiency, the researchers employ LLaMA-Adapter. This method involves inserting learnable "adaptation prompts" into the upper layers of the LLM and utilizing a zero-initialized attention mechanism to preserve pre-trained knowledge during the early stages of training.
The GenTranslate architecture, illustrating the integration of foundation models with an LLM via adapters.

Datasets and Benchmarking

To support this research, the authors introduced HypoTranslate, a large-scale dataset containing over 592,000 hypothesis-translation pairs across 11 languages. This dataset was constructed by decoding $N$-best lists from existing corpora using SeamlessM4T-Large. Evaluation was conducted on several industry-standard benchmarks, including FLEURS, CoVoST-2, WMT, and FLORES.

Key Insights

  • Evidence in the $N$-best List: Through t-SNE visualizations, the authors demonstrated that hypotheses ranked 2nd through $N$th often contain semantic information closer to the ground truth than the top-1 selection. The LLM effectively "cherry-picks" these fragments.
  • Generative Synthesis: GenTranslate is not merely a selection tool; it can generate novel tokens not present in any single hypothesis by reasoning across the provided candidates.
  • Optimal Hypothesis Count: Empirical results suggest that $N=5$ provides the ideal balance of diversity and quality. Larger values of $N$ can introduce noise or redundancy, leading to hallucinations.
  • Reasoning Over Multilingualism: The model's performance is heavily influenced by the LLM's underlying reasoning capabilities (e.g., LLaMA-2's English proficiency) rather than just its multilingual training.

Results

GenTranslate consistently outperforms state-of-the-art (SOTA) baselines across both speech and text modalities:

  • Speech Translation: Achieved a 3.0 BLEU point improvement over SeamlessM4T-Large on the FLEURS (X→En) benchmark, outperforming major models such as AudioPaLM2 and Whisper-Large V2.
  • Machine Translation: Demonstrated a gain of approximately 2.4 BLEU points on WMT (En→X) tasks.

Limitations and Future Work

Current limitations include a dependency on external foundation models for candidate generation and a performance bias toward English as the target language, likely due to the training data of the base LLM (LLaMA-2). Future research aims to integrate the LLM more deeply into the initial translation phase to create a more unified end-to-end system.

Summary updated

Abstract

Recent advances in large language models (LLMs) have stepped forward the development of multilingual speech and machine translation by its reduced representation errors and incorporated external knowledge. However, both translation tasks typically utilize beam search decoding and top-1 hypothesis selection for inference. These techniques struggle to fully exploit the rich information in the diverse N-best hypotheses, making them less optimal for translation tasks that require a single, high-quality output sequence. In this paper, we propose a new generative paradigm for translation tasks, namely GenTranslate, which builds upon LLMs to generate better results from the diverse translation versions in N-best list. Leveraging the rich linguistic knowledge and strong reasoning abilities of LLMs, our new paradigm can integrate the diverse N-best candidates to generate a higher-quality translation result. Furthermore, to support LLM finetuning, we build and release a HypoTranslate dataset that contains over 592K hypotheses-translation pairs in 11 languages. Experiments on various speech and machine translation benchmarks (e.g., FLEURS, CoVoST-2, WMT) demonstrate that our GenTranslate significantly outperforms the state-of-the-art model.
BibTeX
@inproceedings{hu2024gentranslate,
  author = {Yuchen Hu and Chen Chen and Chao-Han Huck Yang and Ruizhe Li and Dong Zhang and Zhehuai Chen and Engsiong Chng},
  title = {{GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.5},
}
Figure