Introduction
GenTranslate is a generative paradigm designed to enhance multilingual speech and machine translation by leveraging the reasoning capabilities of Large Language Models (LLMs). Unlike traditional translation systems that rely on a single "best" output, GenTranslate synthesizes a higher-quality result by integrating information from a list of $N$-best candidate hypotheses.
The Information Loss Problem
Modern translation systems for speech-to-text (ST) and text-to-text (MT) typically utilize beam search decoding. While this process generates multiple candidate hypotheses, standard practice involves discarding all but the top-ranked one (top-1 selection). This approach often leads to significant "information loss," as the top-1 hypothesis may contain minor errors while correct semantic or grammatical fragments are preserved in lower-ranked candidates. GenTranslate addresses this by treating the $N$-best list as a prompt for an LLM, which acts as a master editor to generate a superior final translation.
Methodology: A Second-Pass Generative Paradigm
GenTranslate operates as a "second-pass" system that refines the output of existing foundation models. The workflow consists of three primary stages:
- Candidate Generation: A foundation model, such as Meta’s SeamlessM4T, processes the source input to produce an $N$-best list of hypotheses.
- LLM Integration: These candidates are provided to an LLM alongside a specific instruction to integrate the versions into a higher-quality translation.
- Efficient Fine-tuning: To maintain computational efficiency, the researchers employ LLaMA-Adapter. This method involves inserting learnable "adaptation prompts" into the upper layers of the LLM and utilizing a zero-initialized attention mechanism to preserve pre-trained knowledge during the early stages of training.
Datasets and Benchmarking
To support this research, the authors introduced HypoTranslate, a large-scale dataset containing over 592,000 hypothesis-translation pairs across 11 languages. This dataset was constructed by decoding $N$-best lists from existing corpora using SeamlessM4T-Large. Evaluation was conducted on several industry-standard benchmarks, including FLEURS, CoVoST-2, WMT, and FLORES.
Key Insights
- Evidence in the $N$-best List: Through t-SNE visualizations, the authors demonstrated that hypotheses ranked 2nd through $N$th often contain semantic information closer to the ground truth than the top-1 selection. The LLM effectively "cherry-picks" these fragments.
- Generative Synthesis: GenTranslate is not merely a selection tool; it can generate novel tokens not present in any single hypothesis by reasoning across the provided candidates.
- Optimal Hypothesis Count: Empirical results suggest that $N=5$ provides the ideal balance of diversity and quality. Larger values of $N$ can introduce noise or redundancy, leading to hallucinations.
- Reasoning Over Multilingualism: The model's performance is heavily influenced by the LLM's underlying reasoning capabilities (e.g., LLaMA-2's English proficiency) rather than just its multilingual training.
Results
GenTranslate consistently outperforms state-of-the-art (SOTA) baselines across both speech and text modalities:
- Speech Translation: Achieved a 3.0 BLEU point improvement over SeamlessM4T-Large on the FLEURS (X→En) benchmark, outperforming major models such as AudioPaLM2 and Whisper-Large V2.
- Machine Translation: Demonstrated a gain of approximately 2.4 BLEU points on WMT (En→X) tasks.
Limitations and Future Work
Current limitations include a dependency on external foundation models for candidate generation and a performance bias toward English as the target language, likely due to the training data of the base LLM (LLaMA-2). Future research aims to integrate the LLM more deeply into the initial translation phase to create a more unified end-to-end system.
Summary updated