Co-Segmentation Aided Two-Stream Architecture for Video Captioning

Jayesh Vaidya, Arulkumar Subramaniam, Anurag MittalWACV 2022

TL;DR

This paper introduces a novel two-stream architecture for video captioning that eliminates the need for pre-trained object detectors. By incorporating a co-segmentation inspired attention module, the model can automatically identify and focus on salient regions within video frames, leading to competitive captioning performance on benchmark datasets.

What Problem Does This Paper Address?

Generating human-understandable captions for videos is a challenging task that requires understanding both the visual content and temporal dynamics. While existing methods often leverage pre-trained object detectors to incorporate object-level information, this approach has several drawbacks. Object detectors may not cover all relevant object categories, can introduce biases from their training data, and their performance can be negatively impacted by domain gaps between the detector's training set and the video captioning dataset. This paper aims to address these limitations by proposing a method that can automatically discover and utilize salient regions for captioning without relying on external object detectors.

Methods

The proposed architecture is a two-stream network consisting of:

  • Global Scene Branch (GSB): This branch captures global visual and temporal information using pre-trained 2D (ResNet-101) and 3D (ResNeXt-101) CNNs. The features from these are concatenated and projected to create a global scene representation.
  • Co-segmentation Branch (CoSB): This branch is designed to identify and extract features from salient local regions. It employs a novel Co-segmentation inspired attention module (COSAM), which uses spatial and channel attention to activate common regions and channels across adjacent frames. COSAM is inspired by co-segmentation techniques that aim to find similar objects across images. Following COSAM, a Salient Region Interaction Module (SRIM) uses multi-head self-attention to promote interaction between these identified salient regions across frames.

The GSB and CoSB branches are trained with separate cross-entropy losses for caption generation. To encourage knowledge transfer and ensure that both branches learn similar representations, a KL-divergence loss is used for knowledge distillation, forcing the GSB to learn from the salient region information identified by the CoSB.

Datasets

The proposed model was evaluated on two widely-used benchmark datasets:

  • MSR-VTT: Contains 10,000 video clips with an average of 20 English captions per video.
  • MSVD: Contains 1970 videos with approximately 40 English captions per video.

The paper follows standard data splits for training, validation, and testing for both datasets.

Key Insights and Ideas

  • Eliminating Object Detectors: The core idea is that a model can learn to identify salient regions (objects, persons) necessary for captioning without explicit supervision from pre-trained object detectors. This is achieved by leveraging co-segmentation principles.
  • Co-segmentation for Salient Region Discovery: The paper demonstrates that co-segmentation techniques, typically used for finding similar objects across images, can be adapted to identify important regions within video frames for the task of captioning.
  • Salient Region Interaction: Beyond just identifying salient regions, the SRIM module is crucial for enabling these regions to interact with each other across frames, capturing temporal dynamics and relationships that are vital for generating coherent captions.
  • Knowledge Distillation for Integration: Using KL-divergence loss to distill knowledge from the CoSB to the GSB helps to integrate the learned salient region information into the global scene representation, improving the overall captioning performance.

Key Results

  • The proposed model achieves competitive performance on both MSR-VTT and MSVD datasets, outperforming several state-of-the-art methods, particularly those that do not rely on external object detectors.
  • On the MSVD dataset, the model outperforms on METEOR, ROUGE_L, and CIDEr metrics, demonstrating the effectiveness of the combined GSB and CoSB approach.
  • Qualitative visualizations show that the COSAM module successfully attends to key salient regions such as objects and persons, leading to more detailed and accurate captions compared to baseline methods. For instance, the model can generate more specific descriptions like "a piece of" or "mixing ingredients."
  • Ablation studies confirm the significant contributions of both the COSAM and SRIM modules to the overall performance, with the combination of GSB, COSAM, and SRIM yielding the best results, especially for the CIDEr metric.

Limitations and Future Work

  • MSR-VTT Performance: The paper notes that the moderate performance on MSR-VTT might be due to the dataset's high number of discontinuous clips, which can make it challenging for COSAM to consistently determine salient regions.
  • Future Work: The authors suggest that their approach of learning salient regions end-to-end without external detectors is a promising direction. Future work could explore more advanced co-segmentation techniques or investigate how to further improve the interaction between salient regions. The paper also mentions that the model is better at classifying/recognizing objects important for captioning, hinting at potential for improved object recognition in challenging or uncommon classes.

Summary updated

BibTeX
@inproceedings{vaidya2022co,
  author = {Jayesh Vaidya and Arulkumar Subramaniam and Anurag Mittal},
  title = {{Co-Segmentation Aided Two-Stream Architecture for Video Captioning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure