PolySmart @ TRECVid 2024 Video-To-Text

Jiaxin Wu, Wengyu Zhang, Xiao-Yong Wei, Qing LiarXiv 2024 · DOI · arXiv

Introduction

Video-To-Text (VTT) generation is a critical frontier in artificial intelligence, aiming to bridge the gap between visual perception and natural language. The task involves generating accurate, concise, and contextually relevant descriptions for video sequences, which has profound implications for digital accessibility, automated content indexing, and human-computer interaction. While state-of-the-art Vision-Language Models (VLMs) like LLaVA have demonstrated impressive general-purpose capabilities, they often encounter a "domain gap" when applied to specialized benchmarks like TRECVid. This paper details the PolySmart team's approach to the TRECVid 2024 VTT challenge, focusing on how targeted fine-tuning can overcome the limitations of "vanilla" models.

Methodology

The researchers evaluated three primary modeling approaches to determine the most effective strategy for high-quality caption generation:

  1. Vanilla LLaVA (LV): This baseline utilized the standard LLaVA-v1.5-7b model. In this configuration, the model processed only the middle frame of a video, essentially treating the VTT task as an image-captioning problem.
  2. Vanilla LLaVA-NeXT-Video (LV-V): To test the importance of temporal data, the team used LLaVA-NeXT-Video-7B-DPO, a model architecturally designed to ingest and process entire video sequences.
  3. Fine-tuned LLaVA (LV-FT): This was the authors' primary contribution. They applied a rigorous fine-tuning regimen to the standard LLaVA model to align its descriptive style with the TRECVid requirements.

The fine-tuning process was divided into two essential stages: Vision-Language Alignment, which synchronizes the visual encoder's output with the language model's embedding space, and Visual Instruction Tuning, which teaches the model to follow specific descriptive prompts.

LLaVA Architecture for Video-to-Text

Datasets and Training

The efficacy of the LV-FT model relied on a massive, curated dataset. The team aggregated video-text pairs from previous TRECVid cycles, spanning from VTT16 to VTT23. By extracting five representative frames from each video, they constructed a training set comprising 699,683 frame-text pairs. This large-scale dataset provided the necessary variety and stylistic consistency to train the model on the "who, what, where, and when" of video content. Evaluation was performed on the VTT24 dataset, which includes both a Main task and a Robustness task designed to test model stability.

Results and Discussion

The experimental results provided a clear takeaway: domain-specific data often outweighs architectural complexity. The fine-tuned image-based model (LV-FT) emerged as the superior performer across nearly all primary metrics.

Quantitative Performance

LV-FT achieved a BLEU score of 0.128 and a CIDEr score of 0.712, representing a significant improvement over the vanilla LLaVA (0.101 BLEU / 0.637 CIDEr). Interestingly, while the video-native LV-V model struggled with traditional metrics, it achieved the highest SPICE scores. This suggests that while the video-native model had a strong grasp of semantic relationships, it lacked the stylistic alignment necessary for the VTT task.

Distribution Shift and Style Alignment

One of the most compelling insights from the study was the visualization of the "style shift." By employing t-SNE to map high-dimensional textual features into a 2D space, the authors demonstrated that fine-tuning physically shifted the model's output distribution. Post-fine-tuning, the generated captions overlapped almost perfectly with the target VTT dataset distribution, explaining the dramatic rise in CIDEr scores.

t-SNE visualization of distribution shift after fine-tuning

Qualitative Improvements

Beyond the numbers, the fine-tuned model showed a heightened "attention to detail." It was far more likely to identify specific demographic markers (e.g., "Asian children"), clothing types (e.g., "khaki shirt"), and specific actions or poses (e.g., "lying on the floor") that the vanilla models ignored. This suggests that fine-tuning doesn't just change the style of the output, but also the granularity of the model's visual perception.

Conclusion and Future Work

The PolySmart team's findings underscore the critical role of domain-specific fine-tuning in specialized VLM tasks. The fact that a fine-tuned image model outperformed a vanilla video model suggests that data quality and task alignment are currently more impactful than temporal modeling alone. Due to technical constraints, the team was unable to fine-tune the LLaVA-NeXT-Video model during this cycle. Future research will focus on integrating these fine-tuning strategies with video-native architectures to combine temporal awareness with high-fidelity descriptive capabilities.

Summary updated

Abstract

In this paper, we present our methods and results for the Video-To-Text (VTT) task at TRECVid 2024, exploring the capabilities of Vision-Language Models (VLMs) like LLaVA and LLaVA-NeXT-Video in generating natural language descriptions for video content. We investigate the impact of fine-tuning VLMs on VTT datasets to enhance description accuracy, contextual relevance, and linguistic consistency. Our analysis reveals that fine-tuning substantially improves the model's ability to produce more detailed and domain-aligned text, bridging the gap between generic VLM tasks and the specialized needs of VTT. Experimental results demonstrate that our fine-tuned model outperforms baseline VLMs across various evaluation metrics, underscoring the importance of domain-specific tuning for complex VTT tasks.
BibTeX
@misc{wu2024polysmart,
  author = {Jiaxin Wu and Wengyu Zhang and Xiao-Yong Wei and Qing Li},
  title = {{PolySmart @ TRECVid 2024 Video-To-Text}},
  year = {2024},
  eprint = {2412.15509},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.15509},
}
Figure