Transfer Learning for Sequence Generation: from Single-source to Multi-source

Xuancheng Huang, Jingfang Xu, Maosong Sun, Yang Liu

Abstract

Multi-source sequence generation (MSG) tasks, such as automatic post-editing and multi-source translation, require models to synthesize information from multiple input streams. While large-scale pretrained models like mBART have revolutionized single-source generation, they struggle with MSG due to catastrophic forgetting and inadequate cross-source interaction modeling. We present TRICE (Transferring fRamework for multI-sourCe sEquence generation), a novel framework that bridges this gap through a gradual finetuning strategy and a specialized coarse-to-fine encoder architecture.

Introduction

Multi-source sequence generation (MSG) involves producing a target sequence from multiple source inputs. Examples include automatic post-editing (APE), where a model corrects a machine-translated sentence based on the original source, and multi-document summarization.

Despite the success of pretrained models like BERT and BART, they are primarily designed for single-source data. Standard approaches to MSG often involve simple concatenation of sources, which leads to two primary issues:

  1. Catastrophic Forgetting: The drastic shift from a general pretraining objective to a specialized, low-resource multi-source task causes the model to lose its general linguistic capabilities.
  2. Poor Cross-Source Interaction: Standard self-attention layers are not optimized to capture the nuanced relationships and potential contradictions between different input sources.

The TRICE Framework

Gradual Finetuning

To mitigate catastrophic forgetting, TRICE introduces an intermediate step between pretraining and the final multi-source task. Instead of jumping directly to MSG, the model is first finetuned on a single-source generation (SSG) task using the same languages. This "warm-up" phase reduces the architectural and objective discrepancy, allowing the model to adapt to the generation format before handling multiple inputs.

The TRICE gradual finetuning pipeline: from pretraining to single-source and finally multi-source finetuning.

Coarse-to-Fine Architecture

TRICE employs a hierarchical encoding strategy to better model source interactions. The architecture consists of two main components:

  • Coarse Encoder: This is the original pretrained encoder. It processes concatenated sources to extract general representations, aided by segment embeddings to help the model distinguish between different input languages.
  • Fine Encoder: A newly added, randomly initialized module. It utilizes cross-attention sublayers specifically designed to allow representations from one source to interact with and learn from other sources.
  • Modified Decoder: The decoder implements "separated cross-attention." Rather than attending to a single concatenated representation, it attends to each source independently and combines the results via mean pooling, ensuring that no specific input is ignored during generation.
The detailed TRICE architecture featuring the Coarse Encoder, Fine Encoder with cross-attention, and the modified Decoder.

Experimental Evaluation

The framework was evaluated across three distinct MSG tasks:

  • Automatic Post-Editing (APE): WMT17 English-German SMT task.
  • Multi-source Translation: German and French sources translated into English (WMT14).
  • Document-level Translation: IWSLT2017 and News Commentary, treating preceding context as an additional source.

Results and Discussion

TRICE consistently outperformed strong baselines, including existing pretrained multi-source models like DUALBART. In the APE task, TRICE achieved a new state-of-the-art (SOTA) on WMT17, reaching a 17.41 TER (lower is better) and 73.43 BLEU in low-resource settings. For multi-source translation, it reached 41.5 BLEU on the WMT14 test set, a significant improvement over DUALBART's 40.2 BLEU.

Ablation studies confirmed that the "Fine Encoder" is critical; without explicit cross-attention between sources, performance drops significantly. Furthermore, while the framework adds some computational overhead, the authors found that a limited number of fine layers provides the best balance between performance and efficiency.

Conclusion

TRICE demonstrates that the gap between single-source pretraining and multi-source generation can be effectively bridged. By combining gradual finetuning with a dedicated interaction module (the Fine Encoder), TRICE enables pretrained models to excel in complex multi-input tasks, setting new benchmarks in translation and post-editing.

Summary updated

Abstract

Multi-source sequence generation (MSG) is an important kind of sequence generation tasks that takes multiple sources, including automatic post-editing, multi-source translation, multi-document summarization, etc. As MSG tasks suffer from the data scarcity problem and recent pretrained models have been proven to be effective for low-resource downstream tasks, transferring pretrained sequence-to-sequence models to MSG tasks is essential. Although directly finetuning pretrained models on MSG tasks and concatenating multiple sources into a single long sequence is regarded as a simple method to transfer pretrained models to MSG tasks, we conjecture that the direct finetuning method leads to catastrophic forgetting and solely relying on pretrained self-attention layers to capture cross-source information is not sufficient. Therefore, we propose a two-stage finetuning method to alleviate the pretrain-finetune discrepancy and introduce a novel MSG model with a fine encoder to learn better representations in MSG tasks. Experiments show that our approach achieves new state-of-the-art results on the WMT17 APE task and multi-source translation task using the WMT14 test set. When adapted to document-level translation, our framework outperforms strong baselines significantly.
BibTeX
@inproceedings{huang2021transfer,
  author = {Xuancheng Huang and Jingfang Xu and Maosong Sun and Yang Liu},
  title = {{Transfer Learning for Sequence Generation: from Single-source to Multi-source}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.446},
}
Figure