Abstract
Multi-source sequence generation (MSG) tasks, such as automatic post-editing and multi-source translation, require models to synthesize information from multiple input streams. While large-scale pretrained models like mBART have revolutionized single-source generation, they struggle with MSG due to catastrophic forgetting and inadequate cross-source interaction modeling. We present TRICE (Transferring fRamework for multI-sourCe sEquence generation), a novel framework that bridges this gap through a gradual finetuning strategy and a specialized coarse-to-fine encoder architecture.
Introduction
Multi-source sequence generation (MSG) involves producing a target sequence from multiple source inputs. Examples include automatic post-editing (APE), where a model corrects a machine-translated sentence based on the original source, and multi-document summarization.
Despite the success of pretrained models like BERT and BART, they are primarily designed for single-source data. Standard approaches to MSG often involve simple concatenation of sources, which leads to two primary issues:
- Catastrophic Forgetting: The drastic shift from a general pretraining objective to a specialized, low-resource multi-source task causes the model to lose its general linguistic capabilities.
- Poor Cross-Source Interaction: Standard self-attention layers are not optimized to capture the nuanced relationships and potential contradictions between different input sources.
The TRICE Framework
Gradual Finetuning
To mitigate catastrophic forgetting, TRICE introduces an intermediate step between pretraining and the final multi-source task. Instead of jumping directly to MSG, the model is first finetuned on a single-source generation (SSG) task using the same languages. This "warm-up" phase reduces the architectural and objective discrepancy, allowing the model to adapt to the generation format before handling multiple inputs.
Coarse-to-Fine Architecture
TRICE employs a hierarchical encoding strategy to better model source interactions. The architecture consists of two main components:
- Coarse Encoder: This is the original pretrained encoder. It processes concatenated sources to extract general representations, aided by segment embeddings to help the model distinguish between different input languages.
- Fine Encoder: A newly added, randomly initialized module. It utilizes cross-attention sublayers specifically designed to allow representations from one source to interact with and learn from other sources.
- Modified Decoder: The decoder implements "separated cross-attention." Rather than attending to a single concatenated representation, it attends to each source independently and combines the results via mean pooling, ensuring that no specific input is ignored during generation.
Experimental Evaluation
The framework was evaluated across three distinct MSG tasks:
- Automatic Post-Editing (APE): WMT17 English-German SMT task.
- Multi-source Translation: German and French sources translated into English (WMT14).
- Document-level Translation: IWSLT2017 and News Commentary, treating preceding context as an additional source.
Results and Discussion
TRICE consistently outperformed strong baselines, including existing pretrained multi-source models like DUALBART. In the APE task, TRICE achieved a new state-of-the-art (SOTA) on WMT17, reaching a 17.41 TER (lower is better) and 73.43 BLEU in low-resource settings. For multi-source translation, it reached 41.5 BLEU on the WMT14 test set, a significant improvement over DUALBART's 40.2 BLEU.
Ablation studies confirmed that the "Fine Encoder" is critical; without explicit cross-attention between sources, performance drops significantly. Furthermore, while the framework adds some computational overhead, the authors found that a limited number of fine layers provides the best balance between performance and efficiency.
Conclusion
TRICE demonstrates that the gap between single-source pretraining and multi-source generation can be effectively bridged. By combining gradual finetuning with a dedicated interaction module (the Fine Encoder), TRICE enables pretrained models to excel in complex multi-input tasks, setting new benchmarks in translation and post-editing.
Summary updated