Introduction
Video-To-Text (VTT) generation is a critical frontier in artificial intelligence, aiming to bridge the gap between visual perception and natural language. The task involves generating accurate, concise, and contextually relevant descriptions for video sequences, which has profound implications for digital accessibility, automated content indexing, and human-computer interaction. While state-of-the-art Vision-Language Models (VLMs) like LLaVA have demonstrated impressive general-purpose capabilities, they often encounter a "domain gap" when applied to specialized benchmarks like TRECVid. This paper details the PolySmart team's approach to the TRECVid 2024 VTT challenge, focusing on how targeted fine-tuning can overcome the limitations of "vanilla" models.
Methodology
The researchers evaluated three primary modeling approaches to determine the most effective strategy for high-quality caption generation:
- Vanilla LLaVA (LV): This baseline utilized the standard LLaVA-v1.5-7b model. In this configuration, the model processed only the middle frame of a video, essentially treating the VTT task as an image-captioning problem.
- Vanilla LLaVA-NeXT-Video (LV-V): To test the importance of temporal data, the team used LLaVA-NeXT-Video-7B-DPO, a model architecturally designed to ingest and process entire video sequences.
- Fine-tuned LLaVA (LV-FT): This was the authors' primary contribution. They applied a rigorous fine-tuning regimen to the standard LLaVA model to align its descriptive style with the TRECVid requirements.
The fine-tuning process was divided into two essential stages: Vision-Language Alignment, which synchronizes the visual encoder's output with the language model's embedding space, and Visual Instruction Tuning, which teaches the model to follow specific descriptive prompts.
Datasets and Training
The efficacy of the LV-FT model relied on a massive, curated dataset. The team aggregated video-text pairs from previous TRECVid cycles, spanning from VTT16 to VTT23. By extracting five representative frames from each video, they constructed a training set comprising 699,683 frame-text pairs. This large-scale dataset provided the necessary variety and stylistic consistency to train the model on the "who, what, where, and when" of video content. Evaluation was performed on the VTT24 dataset, which includes both a Main task and a Robustness task designed to test model stability.
Results and Discussion
The experimental results provided a clear takeaway: domain-specific data often outweighs architectural complexity. The fine-tuned image-based model (LV-FT) emerged as the superior performer across nearly all primary metrics.
Quantitative Performance
LV-FT achieved a BLEU score of 0.128 and a CIDEr score of 0.712, representing a significant improvement over the vanilla LLaVA (0.101 BLEU / 0.637 CIDEr). Interestingly, while the video-native LV-V model struggled with traditional metrics, it achieved the highest SPICE scores. This suggests that while the video-native model had a strong grasp of semantic relationships, it lacked the stylistic alignment necessary for the VTT task.
Distribution Shift and Style Alignment
One of the most compelling insights from the study was the visualization of the "style shift." By employing t-SNE to map high-dimensional textual features into a 2D space, the authors demonstrated that fine-tuning physically shifted the model's output distribution. Post-fine-tuning, the generated captions overlapped almost perfectly with the target VTT dataset distribution, explaining the dramatic rise in CIDEr scores.
Qualitative Improvements
Beyond the numbers, the fine-tuned model showed a heightened "attention to detail." It was far more likely to identify specific demographic markers (e.g., "Asian children"), clothing types (e.g., "khaki shirt"), and specific actions or poses (e.g., "lying on the floor") that the vanilla models ignored. This suggests that fine-tuning doesn't just change the style of the output, but also the granularity of the model's visual perception.
Conclusion and Future Work
The PolySmart team's findings underscore the critical role of domain-specific fine-tuning in specialized VLM tasks. The fact that a fine-tuned image model outperformed a vanilla video model suggests that data quality and task alignment are currently more impactful than temporal modeling alone. Due to technical constraints, the team was unable to fine-tune the LLaVA-NeXT-Video model during this cycle. Future research will focus on integrating these fine-tuning strategies with video-native architectures to combine temporal awareness with high-fidelity descriptive capabilities.
Summary updated