TL;DR
This paper introduces GraMMo, a novel framework for generating concise video headlines by "grafting" pre-trained language and video-text models. This approach effectively leverages existing powerful models to overcome the challenge of limited annotated data for multimodal headline generation, achieving state-of-the-art results.
What Problem Does This Paper Address?
Generating compelling headlines for short videos is crucial for quickly conveying their content. This task, known as multimodal headline generation, requires understanding both the visual information from the video and the textual information from its transcript. However, creating large, manually annotated datasets for this specific task is labor-intensive and impractical, hindering the development of effective models. While pre-trained language models (PLMs) and video-language models have shown promise in related areas, directly applying them to multimodal headline generation is challenging due to the need for modality balance – effectively combining visual and textual cues.
Methods
The core innovation of GraMMo lies in its "grafting" mechanism. Instead of training a multimodal model from scratch, GraMMo leverages pre-trained models for two key tasks: language generation (like text summarization) and video-text matching.
- Grafting Architecture: GraMMo comprises a video encoder, a text encoder, and a text decoder. The key idea is to initialize these components by "grafting" pre-trained models. Specifically, a pre-trained language model (like PALM) provides the text encoder and decoder, while a pre-trained video-text matching model contributes the video encoder.
- Joint-Modality Layer: To effectively fuse information from the video and text modalities, a novel "joint-modality layer" is introduced. This layer employs a two-way attention strategy to capture both the commonalities and specificities of the video and text. It generates a "video-enhanced text feature" by allowing text tokens to attend to relevant video content and a "video-specific feature" that complements the text.
- Pre-training Tasks:
- Language Generation: The text encoder and decoder are pre-trained on a large corpus for abstractive text summarization, using masked word prediction and generation loss.
- Video-Text Understanding: The video encoder is pre-trained using a video-text matching task on a large dataset of video-text pairs. Contrastive loss (InfoNCE) is used to align video and text embeddings.
- Fine-tuning: After grafting, the entire model is fine-tuned on a smaller, task-specific multimodal headline generation dataset.
Datasets
- Pre-training Datasets:
- NLG: A 14GB high-quality Chinese corpus (CLUE-small) comprising News, WebText, Wikipedia, and Comments.
- Video-Text Matching: 3.2 million videos from a Chinese video platform, with headlines provided by uploaders.
- Fine-tuning Dataset:
- WB-News: A newly established dataset for multimodal headline generation, containing over 43,000 samples collected from official Chinese media Weibo accounts. Each sample includes video, source transcript, and target summary.
Key Insights and Ideas
- Leveraging Pre-trained Models is Key: The paper strongly advocates for "grafting" pre-trained models as a highly effective strategy to overcome data scarcity in multimodal tasks. This significantly reduces the need for large, manually annotated datasets.
- Modality Balance is Crucial: Simply concatenating features from different modalities is insufficient. The proposed joint-modality layer, with its two-way attention mechanism, is vital for capturing nuanced interactions between video and text, leading to better headline generation.
- Video Modality Enhances Text Summaries: The experiments demonstrate that incorporating video information significantly improves headline generation quality compared to text-only models, even outperforming strong text summarization baselines.
- Faster Convergence: Fine-tuning a grafted model converges much faster than training a model from scratch, making deployment more efficient.
Key Results
- State-of-the-Art Performance: GraMMo achieved the best results on the WB-News dataset, outperforming existing multimodal generation methods and strong text-only models like PALM by a significant margin.
- Improved Metrics: The paper reports improvements across various metrics, including ROUGE, BLEU, and METEOR. For instance, GraMMo achieved an R-L score of 35.38, compared to PALM's 33.86.
- Effective Video-Text Alignment: The video-text retrieval experiments showed good alignment, with R@1 and R@5 scores of approximately 40% and 60%, respectively, indicating that the model effectively maps video and text into a common semantic space.
- Human Evaluation: GraMMo also performed better in human evaluations for readability and informativeness compared to PALM and MMPT.
Limitations and Future Work
- Fact Inconsistency and Incomplete Information: While GraMMo improves informativeness, the paper acknowledges that fact inconsistency and incomplete key information remain challenges in generated headlines, which will be investigated in future work.
- GraMMo-Video Limitations: The GraMMo-Video model (using only video input) struggled with factual information, highlighting the importance of text for precise details. However, it showed an ability to capture topic semantics.
- Future Directions: The authors suggest further work on improving the quality of generated headlines by addressing fact inconsistency and information completeness. The proposed framework is also noted as being extensible to other multimodal tasks.
Summary updated