Domain-Specific Semantics Guided Approach to Video Captioning

Hemalatha M, C Chandra ShekharWACV 2020

TL;DR

This paper proposes a novel approach to video captioning that leverages domain-specific knowledge to generate more accurate and relevant descriptions. By classifying videos into domains and training specialized decoders for each, the model significantly improves captioning performance, especially for videos with nuanced content.

What Problem Does This Paper Address?

Video captioning, the task of generating a natural language description for a video, is a challenging problem. While image captioning has seen significant progress, video captioning is more complex due to the temporal dimension. Existing methods often treat all videos uniformly, failing to account for the fact that videos belong to specific domains (e.g., sports, cooking, music) and that domain knowledge is crucial for accurate description. For instance, describing a cooking video requires different vocabulary and understanding than describing a sports video. Current approaches that do consider domain information often rely on limited training data or struggle to generalize, leading to suboptimal performance. This paper aims to bridge this gap by explicitly incorporating domain-specific semantics into the captioning process.

Methods

The proposed approach consists of several key components:

  • Domain Classification: The system first classifies videos into different domains. This is achieved by extracting spatial (2D-CNN) and spatio-temporal (3D-CNN) features, which are then aggregated using the Temporal VLAD method. These aggregated features, along with semantic features extracted from video captions, are fed into a multi-class classifier (initially using K-means for pseudo-labeling on MSVD, and category labels on MSR-VTT). This classifier outputs probabilities for each domain.
  • Domain-Specific Decoders: Based on the domain classification, the top-k most probable domains are identified for each video. The system then uses a novel method to train domain-specific decoders. Crucially, videos are shared across multiple domains for training if they fall into the top-k predicted domains. This allows the decoders to learn domain-specific vocabulary and semantic tags from a richer set of training data, mitigating the issue of limited data per domain.
  • Temporal VLAD Preprocessing: To improve feature representation, the paper employs Temporal VLAD. This method aggregates frame-level descriptors (from 2D-CNN and 3D-CNN) into a compact video-level representation by clustering the descriptors and accumulating differences from cluster centers. This is applied to both spatial and spatio-temporal features.
  • Domain-Specific Semantics: The domain-specific decoders utilize vocabulary derived from the captions of training videos within their respective domains. This ensures that the language model is tailored to the specific content and terminology of each domain.

During testing, captions are generated by all domain-specific decoders, and the sentence with the highest probability is selected as the final output.

Datasets

The paper evaluates its approach on two widely used video captioning datasets:

  • Microsoft Video Description Corpus (MSVD): This dataset contains approximately 1970 YouTube video clips with about 40 annotations per clip, totaling around 80,839 annotations. The authors use a split of 1200 clips for training, 100 for validation, and 670 for testing.
  • MSR-VTT (MSR Video to Text): This is a larger dataset with 10,000 videos, each with 20 human-annotated descriptions. Videos are categorized into 20 types (e.g., music, people, gaming). The authors use the standard MM2016 challenge split: 6513 videos for training, 2990 for testing, and 497 for validation.

Key Insights and Ideas

  • Domain Matters: The core insight is that domain-specific knowledge significantly enhances video captioning. Treating all videos as belonging to a single, generic domain overlooks crucial contextual information.
  • Sharing is Caring (for Decoders): The strategy of sharing videos across multiple top-k predicted domains for training domain-specific decoders is a clever way to combat data sparsity within individual domains, leading to more robust models.
  • VLAD for Better Features: Temporal VLAD proves effective in creating more discriminative video representations by aggregating frame-level features, leading to improved performance.
  • Soft Assignment for Multi-Domain Videos: Assigning videos to multiple domains (via soft assignment) rather than a single hard label better reflects the reality of videos that might span multiple categories.

Key Results

The proposed approach, particularly the "DSD Top-3 DS-SEM" (Domain-Specific Decoder Top-3 with Domain-Specific Semantics), demonstrates superior performance across various metrics (BLEU-4, METEOR, CIDEr, RG-L) on both MSVD and MSR-VTT datasets compared to baseline methods and state-of-the-art approaches.

  • MSVD: The DSD Top-3 DS-SEM achieved scores of 50.1 (BL-4), 34.7 (METEOR), 76.0 (CIDr), and 73.1 (RG-L).
  • MSR-VTT: The DSD Top-3 DS-SEM achieved scores of 45.2 (BL-4), 29.9 (METEOR), 51.1 (CIDr), and 64.2 (RG-L).

The paper highlights that VLAD aggregation improves performance over models without it. Furthermore, domain-specific decoders significantly outperform models without them. The "DSD Top-1" model performed worse than "Top-3" and "Top-5," suggesting that limiting a video to a single domain for training can lead to overfitting. The use of domain-specific semantics further boosted performance.

Limitations and Future Work

  • Data Scarcity for Niche Domains: The paper notes that for domains with very few training videos (like "technology" in their examples), the performance of the domain-specific decoder can still be suboptimal, even with overlap.
  • Future Work: The authors suggest that further research could explore more sophisticated methods for domain classification and decoder training, especially for domains with limited data. The paper also implicitly points towards the need for larger, more diverse datasets that better represent the vast spectrum of video content.

Summary updated

BibTeX
@inproceedings{m2020domain,
  author = {Hemalatha M and C Chandra Shekhar},
  title = {{Domain-Specific Semantics Guided Approach to Video Captioning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure