CALVIN: Improved Contextual Video Captioning via Instruction Tuning

Gowthami Somepalli, Arkabandhu Chowdhury, Jonas Geiping, Ronen Basri, Tom Goldstein, David W. Jacobs

Imagine watching a movie clip where a man hands a letter to a woman. A standard video captioning model might describe the scene as, "A man in a suit hands a piece of paper to a woman." While technically accurate, this description is virtually useless for someone relying on audio descriptions (AD) because it lacks narrative context. Who are these characters? Is this a crucial plot point?

Current Vision-Language Models (VLMs) struggle with video captioning due to two main issues: verbosity and superficiality (focusing on background objects rather than narrative action) and context blindness (treating each clip as an isolated event without recognizing established characters or plotlines).

To bridge the gap between literal "image-stringing" and true contextual storytelling, researchers introduce CALVIN, a specialized video Large Language Model (LLM) designed to generate concise, narrative-driven scene descriptions for movies.

Architecture and Training Methodology

CALVIN's architecture is built on three core components: a frozen CLIP ViT-h/14 vision encoder, a trainable Q-Former that acts as a bridge, and a Llama-2 7B LLM.

To achieve its contextual awareness, CALVIN undergoes a two-stage training process:

  1. Stage 1 (Alignment): The projection module (Q-Former and a linear layer) is trained on massive image-caption datasets. This aligns the visual features extracted by the encoder with the LLM's text space.
  2. Stage 2 (Instruction Tuning): The model is fine-tuned on a mixture of high-quality image-VQA and video captioning data. Crucially, when processing movie data, CALVIN is fed text descriptions of preceding scenes as "context tokens."
CALVIN's two-stage training architecture aligns vision and language before fine-tuning the model to incorporate context tokens for narrative captioning.

Unlike previous approaches that required a separate third stage for movie-specific training, CALVIN integrates this into the second stage, simplifying the pipeline and accelerating convergence.

Key Insights: The Power of Context

CALVIN's training yields several fascinating insights into how VLMs process narrative video:

  • Context is a Hallucination Killer: Without prior context, models often hallucinate or guess character names (e.g., misidentifying "Valjean" as "Victor"). Providing CALVIN with just three previous scenes of context drastically reduces these errors, resulting in captions that are accurate and narrative-focused.
  • Image-VQA is the "Secret Sauce": Surprisingly, mixing image-based question-answering data into the video training phase significantly enhances the model's overall visual reasoning capabilities, even for video-only tasks.
  • Few-Shot Adaptability: CALVIN can be personalized to a specific movie at test time. By exposing the model to just 20-50 scenes from a new film, it quickly learns the specific character names and the unique descriptive "tone" of that movie.
Qualitative comparisons demonstrate how CALVIN-3S (using 3 scenes of context) correctly identifies characters like Valjean and Cosette, producing more accurate and concise narrations than baseline models.

State-of-the-Art Results

CALVIN achieves state-of-the-art (SOTA) performance on the MAD-eval benchmark, a primary dataset for contextual movie audio descriptions. Compared to the previous best model (AutoAD-II), CALVIN demonstrated a ~26% improvement in CIDEr and a ~68% improvement in BertScore.

Furthermore, in zero-shot evaluations on the TV Captioning (TVC) dataset—meaning the model was never explicitly trained on this data—CALVIN outperformed popular off-the-shelf models like VideoLlama and VideoLLaVA by a wide margin. In the CIDEr metric alone, CALVIN was nearly three times better than the runner-up.

Qualitatively, CALVIN's captions are notably crisper. Instead of rambling about background details, it focuses on plot-relevant actions, delivering descriptions like "Valjean strokes her hair" rather than generic, disconnected observations.

Limitations and Future Directions

While CALVIN represents a significant leap forward in contextual video captioning, the authors note a few areas for future improvement:

  • Data Scarcity: There remains a pressing need for more high-quality, publicly available video datasets with rich, narrative-driven annotations.
  • Long-Form Video Processing: Currently, the model samples a limited number of frames. Processing longer sequences or employing more intelligent frame-sampling techniques could help capture finer narrative details.
  • Multimodality: Future iterations could greatly benefit from incorporating audio streams (such as dialogue and sound effects) to further enrich the model's contextual understanding of a scene.

Summary updated

Abstract

The recent emergence of powerful Vision-Language models (VLMs) has significantly improved image captioning. Some of these models are extended to caption videos as well. However, their capabilities to understand complex scenes are limited, and the descriptions they provide for scenes tend to be overly verbose and focused on the superficial appearance of objects. Scene descriptions, especially in movies, require a deeper contextual understanding, unlike general-purpose video captioning. To address this challenge, we propose a model, CALVIN, a specialized video LLM that leverages previous movie context to generate fully "contextual" scene descriptions. To achieve this, we train our model on a suite of tasks that integrate both image-based question-answering and video captioning within a unified framework, before applying instruction tuning to refine the model's ability to provide scene captions. Lastly, we observe that our model responds well to prompt engineering and few-shot in-context learning techniques, enabling the user to adapt it to any new movie with very little additional annotation.
BibTeX
@inproceedings{somepalli2024calvin,
  author = {Gowthami Somepalli and Arkabandhu Chowdhury and Jonas Geiping and Ronen Basri and Tom Goldstein and David W. Jacobs},
  title = {{CALVIN: Improved Contextual Video Captioning via Instruction Tuning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=7Kz7icCZ6H},
}
Figure