Grafting Pre-trained Models for Multimodal Headline Generation

Lingfeng Qiao, Chen Wu, Ye Liu, Haoyuan Peng, Di Yin, Bo Ren

In the era of short-form video, a catchy headline is the difference between a viral hit and digital dust. But generating these headlines automatically is a complex "multimodal" challenge: a model must understand both the visual action and the spoken transcript to produce a concise, readable summary.

Researchers from Tencent Youtu Lab have proposed a clever solution in their paper, "Grafting Pre-trained Models for Multimodal Headline Generation." By "grafting" together specialized pre-trained models, they’ve created a system called GraMMo that outperforms existing methods while requiring significantly less specialized training data.

TL;DR

GraMMo creates a multimodal headline generator by "grafting" a pre-trained video encoder onto a pre-trained generative language model. This approach solves the problem of scarce annotated video-summary data and uses a novel fusion layer to balance visual and textual information, achieving state-of-the-art results on real-world news datasets.

What Problem Does This Paper Address?

Generating headlines for videos is harder than summarizing text. You need to capture the "core" of the video, which might be hidden in the frames, the transcript, or the interaction between both.

The field faces three major hurdles:

  1. Data Scarcity: Large-scale "triplet" datasets (video + transcript + human-written headline) are incredibly expensive and labor-intensive to create.
  2. Architecture Mismatch: Most pre-trained models (PTMs) are either great at understanding (like CLIP for video-text matching) or great at generating (like GPT or BART for text), but rarely both for multimodal inputs.
  3. The Modality Balance: Simply "gluing" video and text features together often leads to the model ignoring one modality or getting overwhelmed by redundant information.
Figure 1: An example of multimodal generation.

Methods

The authors introduce GraMMo (Grafting Multimodal generation model). Instead of training a massive multimodal model from scratch, they use a botanical metaphor: grafting.

1. The Grafting Process

They take high-performing "limbs" from existing models:

  • Video Encoder: Grafted from a model pre-trained on video-text matching (learning to align videos with descriptions).
  • Text Encoder & Decoder: Grafted from a generative language model (PALM), pre-trained on massive text corpora for summarization.

2. The Joint-Modality Layer

This is the "connective tissue" of the graft. To prevent the model from ignoring the video or getting confused by redundant text, they use a two-way attention strategy:

  • Video-Enhanced Text Features: Uses the video to highlight the most important tokens in the transcript.
  • Video-Specific Features: This is the clever bit. The model explicitly identifies visual information that is not present in the text and adds it as a supplement.

3. Dynamic Frame Sampling (DFS)

To handle videos of different lengths efficiently, they use a DFS layer that chooses the most representative frames to convert into video tokens.

Figure 2: GraMMo framework for multimodal headline generation.

Datasets

  • Pre-training: The language model used 14GB of Chinese text (CLUE-small), and the video-text matching used 3.2 million video-description pairs.
  • WB-News (New Benchmark): The authors introduced a new dataset of over 43,000 triplets collected from official Weibo news accounts. Each sample includes a video, a transcript (ASR), and a human-written headline.

Key Insights and Ideas

  • Specialization Beats Generalization: Grafting specialized models (one for seeing, one for speaking) is more effective than trying to train one model to do everything from scratch when data is limited.
  • Focus on the "Specialty": The most important part of multimodal fusion isn't just finding where the video and text agree, but finding the "video-specific" information that the text missed.
  • Efficiency: Because the "limbs" are already "grown" (pre-trained), the model converges much faster during fine-tuning.

Key Results

GraMMo was tested against several state-of-the-art (SOTA) multimodal models like MV-GPT and MMPT.

  • Performance: GraMMo achieved the highest scores across all major metrics (ROUGE-L, BLEU-4, and METEOR). For instance, it hit a 35.38 ROUGE-L, significantly outperforming text-only baselines.
  • Training Speed: As shown in the loss curves, the grafted model starts with a much lower loss and converges far more rapidly than a model trained from scratch.
  • Human Evaluation: In blind tests, humans rated GraMMo's headlines as more readable and informative than those generated by other SOTA models.
Figure 3: Different initialization methods.

Limitations and Future Work

While GraMMo is a significant step forward, the authors acknowledge a few remaining challenges:

  • Fact Inconsistency: Like many generative models, it occasionally hallucinates or gets specific facts (like names or dates) wrong.
  • Incomplete Information: Sometimes the model still misses key details from the source material.
  • Future Directions: The team plans to investigate ways to improve factual consistency and ensure more complete information extraction in future iterations.

The GraMMo approach proves that in the world of AI, sometimes the best way to build a powerful new system isn't to start from a seed, but to graft the best existing branches together.

Summary updated

Abstract

Multimodal headline utilizes both video frames and transcripts to generate the natural language title of the videos. Due to a lack of large-scale, manually annotated data, the task of annotating grounded headlines for video is labor intensive and impractical. Previous researches on pre-trained language models and video-language models have achieved significant progress in related downstream tasks. However, none of them can be directly applied to multimodal headline architecture where we need both multimodal encoder and sentence decoder. A major challenge in simply gluing language model and video-language model is the modality balance, which is aimed at combining visual-language complementary abilities. In this paper, we propose a novel approach to graft the video encoder from the pre-trained video-language model on the generative pre-trained language model. We also present a consensus fusion mechanism for the integration of different components, via inter/intra modality relation. Empirically, experiments show that the grafted model achieves strong results on a brand-new dataset collected from real-world applications.
BibTeX
@inproceedings{qiao2022grafting,
  author = {Lingfeng Qiao and Chen Wu and Ye Liu and Haoyuan Peng and Di Yin and Bo Ren},
  title = {{Grafting Pre-trained Models for Multimodal Headline Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=_FDT9yQ65Mz},
}
Figure