TL;DR
This paper introduces BERTGEN, a novel generative model that extends BERT's capabilities to handle multimodal and multilingual generation tasks. By fusing pre-trained models like VL-BERT and M-BERT, BERTGEN achieves strong performance across image captioning, machine translation, and multimodal machine translation, even in zero-shot settings, demonstrating the power of multi-task learning and knowledge transfer.
What Problem Does This Paper Address?
While pre-trained language models like BERT have revolutionized natural language understanding (NLU), their application to natural language generation (NLG) tasks has been less explored. Traditional approaches often require task-specific fine-tuning, and some argue that BERT's masked language modeling (MLM) objective is ill-suited for generation. Existing work has focused on using pre-trained models to initialize encoder-decoder architectures or distill knowledge. This paper addresses the challenge of creating a single, versatile generative model that can handle diverse tasks (image captioning, translation) and multiple languages without task-specific fine-tuning, leveraging the strengths of pre-trained multimodal and multilingual models.
Methods
BERTGEN is a decoder-only model that builds upon the BERT architecture. Its key innovations include:
- Hybrid Initialization: BERTGEN is initialized by combining weights from VL-BERT (for visual modality awareness) and M-BERT (for multilingual capabilities). This hybrid approach aims to prime the model for both vision and language tasks across multiple languages.
- Multi-task Learning: The model is trained simultaneously on various generative tasks, including image captioning (IC), machine translation (MT), and multimodal machine translation (MMT). This multi-task approach is hypothesized to encourage the transfer of relevant inductive biases.
- Sequence Unrolling: To handle generation within a unified MLM framework, BERTGEN "unrolls" the generation process. Each token prediction is treated as a separate training example, effectively augmenting the dataset and allowing the model to learn conditional probabilities without explicit encoder-decoder separation.
- Unified Input Representation: Inputs from different modalities (text, images) and languages are represented as a single sequence of embeddings, allowing the Transformer to attend to all inputs bidirectionally. Target language specifiers are used to control output language during generation.
- Zero-Shot Generation: The model is evaluated on its ability to perform tasks it hasn't been explicitly trained on, demonstrating its generalization capabilities.
Datasets
The paper uses a variety of datasets for training and evaluation:
- Image Captioning:
- FLICKR30K (English)
- MULTI30K (German captions for FLICKR30K images)
- TASVIRET (Turkish captions for FLICKR8K images)
- COCO (for evaluation, not training)
- Machine Translation:
- IWSLT'14 (English-German, English-French)
- SETIMES2 (English-Turkish)
- Multimodal Machine Translation:
- MULTI30K (English-German, English-French)
- Zero-Shot Evaluation:
- WMT'19 test set (for DE-FR and FR-DE translation)
Key Insights and Ideas
- Hybrid Initialization is Key: Combining VL-BERT and M-BERT provides a strong foundation for a model that needs to understand both visual and linguistic information across multiple languages. This suggests that leveraging existing specialized pre-trained models is more effective than training from scratch.
- Multi-task Learning Enhances Generalization: Training BERTGEN on a diverse set of tasks significantly improves its performance and its ability to generalize to unseen tasks (zero-shot learning). This indicates that learning multiple related tasks together imbues the model with more robust and transferable knowledge.
- Unified Generation Framework Works: The "sequence unrolling" approach, which adapts the MLM objective for generation, allows for a single, parameter-efficient model that can handle both encoding and decoding without separate components. This simplifies the architecture and potentially reduces computational overhead.
- BERT's Inductive Biases are Transferable: The paper demonstrates that BERT's inherent inductive biases, when combined with multimodal and multilingual pre-training, can be effectively transferred to generative tasks, even those requiring cross-modal understanding.
Key Results
- Outperforms Baselines: BERTGEN consistently outperforms strong baselines across image captioning, machine translation, and multimodal machine translation tasks. For instance, it achieves higher BLEU and METEOR scores than dedicated FAIRSEQ NMT systems on text-only MT.
- Strong Zero-Shot Performance: BERTGEN shows surprising zero-shot capabilities, performing well on unseen translation directions (e.g., DE→FR, FR→DE) and image captioning tasks it wasn't trained on. For example, it achieves competitive scores on zero-shot French captioning.
- Parameter Efficiency: The unified architecture, achieved through sequence unrolling, results in a parameter-efficient model (~89.3M parameters) compared to task-specific models.
- Learning Efficiency: BERTGEN shows faster learning convergence on MT tasks compared to single-task FAIRSEQ systems, often achieving good performance after only a few passes over the training data.
- Robustness to Modality Mismatch: In multimodal machine translation, BERTGEN's performance drops when image-source caption mappings are shuffled, indicating that it effectively utilizes visual information and doesn't simply ignore it.
Limitations and Future Work
- Limitations:
- The paper notes that for zero-shot French captioning, the lack of human references makes metric evaluation challenging, and the scores might be penalized.
- While BERTGEN performs well, it doesn't always achieve state-of-the-art (SOTA) results compared to highly specialized models that might use more training data or specific architectural tweaks.
- The training process, especially with batch size 1, can be computationally intensive (e.g., two days per epoch on a V100 GPU).
- Future Work:
- The authors suggest exploring more sophisticated pre-trained models beyond M-BERT and VL-BERT.
- Further investigation into the factors contributing to BERTGEN's efficient learning, such as sequence unrolling and multi-tasking, is recommended.
- The paper implies that with further optimization and mixed-precision multi-GPU training, training time can be substantially reduced.
Summary updated