In the era of short-form video, a catchy headline is the difference between a viral hit and digital dust. But generating these headlines automatically is a complex "multimodal" challenge: a model must understand both the visual action and the spoken transcript to produce a concise, readable summary.
Researchers from Tencent Youtu Lab have proposed a clever solution in their paper, "Grafting Pre-trained Models for Multimodal Headline Generation." By "grafting" together specialized pre-trained models, they’ve created a system called GraMMo that outperforms existing methods while requiring significantly less specialized training data.
TL;DR
GraMMo creates a multimodal headline generator by "grafting" a pre-trained video encoder onto a pre-trained generative language model. This approach solves the problem of scarce annotated video-summary data and uses a novel fusion layer to balance visual and textual information, achieving state-of-the-art results on real-world news datasets.
What Problem Does This Paper Address?
Generating headlines for videos is harder than summarizing text. You need to capture the "core" of the video, which might be hidden in the frames, the transcript, or the interaction between both.
The field faces three major hurdles:
- Data Scarcity: Large-scale "triplet" datasets (video + transcript + human-written headline) are incredibly expensive and labor-intensive to create.
- Architecture Mismatch: Most pre-trained models (PTMs) are either great at understanding (like CLIP for video-text matching) or great at generating (like GPT or BART for text), but rarely both for multimodal inputs.
- The Modality Balance: Simply "gluing" video and text features together often leads to the model ignoring one modality or getting overwhelmed by redundant information.
Methods
The authors introduce GraMMo (Grafting Multimodal generation model). Instead of training a massive multimodal model from scratch, they use a botanical metaphor: grafting.
1. The Grafting Process
They take high-performing "limbs" from existing models:
- Video Encoder: Grafted from a model pre-trained on video-text matching (learning to align videos with descriptions).
- Text Encoder & Decoder: Grafted from a generative language model (PALM), pre-trained on massive text corpora for summarization.
2. The Joint-Modality Layer
This is the "connective tissue" of the graft. To prevent the model from ignoring the video or getting confused by redundant text, they use a two-way attention strategy:
- Video-Enhanced Text Features: Uses the video to highlight the most important tokens in the transcript.
- Video-Specific Features: This is the clever bit. The model explicitly identifies visual information that is not present in the text and adds it as a supplement.
3. Dynamic Frame Sampling (DFS)
To handle videos of different lengths efficiently, they use a DFS layer that chooses the most representative frames to convert into video tokens.
Datasets
- Pre-training: The language model used 14GB of Chinese text (CLUE-small), and the video-text matching used 3.2 million video-description pairs.
- WB-News (New Benchmark): The authors introduced a new dataset of over 43,000 triplets collected from official Weibo news accounts. Each sample includes a video, a transcript (ASR), and a human-written headline.
Key Insights and Ideas
- Specialization Beats Generalization: Grafting specialized models (one for seeing, one for speaking) is more effective than trying to train one model to do everything from scratch when data is limited.
- Focus on the "Specialty": The most important part of multimodal fusion isn't just finding where the video and text agree, but finding the "video-specific" information that the text missed.
- Efficiency: Because the "limbs" are already "grown" (pre-trained), the model converges much faster during fine-tuning.
Key Results
GraMMo was tested against several state-of-the-art (SOTA) multimodal models like MV-GPT and MMPT.
- Performance: GraMMo achieved the highest scores across all major metrics (ROUGE-L, BLEU-4, and METEOR). For instance, it hit a 35.38 ROUGE-L, significantly outperforming text-only baselines.
- Training Speed: As shown in the loss curves, the grafted model starts with a much lower loss and converges far more rapidly than a model trained from scratch.
- Human Evaluation: In blind tests, humans rated GraMMo's headlines as more readable and informative than those generated by other SOTA models.
Limitations and Future Work
While GraMMo is a significant step forward, the authors acknowledge a few remaining challenges:
- Fact Inconsistency: Like many generative models, it occasionally hallucinates or gets specific facts (like names or dates) wrong.
- Incomplete Information: Sometimes the model still misses key details from the source material.
- Future Directions: The team plans to investigate ways to improve factual consistency and ensure more complete information extraction in future iterations.
The GraMMo approach proves that in the world of AI, sometimes the best way to build a powerful new system isn't to start from a seed, but to graft the best existing branches together.
Summary updated