FDU Participation in TRECVID 2019 VTT Task

Shaoxiang Chen, Yu-Gang Jiang

Abstract

This paper describes the system developed by Fudan University (FDU) for the TRECVID 2019 Video to Text (VTT) task. The approach addresses the semantic gap between visual features and natural language by introducing an auxiliary concept prediction module. By explicitly identifying objects and actions and feeding these latent semantic features into an attention-based LSTM, the model achieved a 3rd-place ranking in the competition.

Introduction

Video captioning involves the complex task of translating a sequence of visual frames into a coherent natural language description. While standard encoder-decoder architectures—typically combining Convolutional Neural Networks (CNNs) for visual encoding and Recurrent Neural Networks (RNNs) for sequence generation—have shown promise, they often struggle with the "domain gap." This gap exists between high-level visual feature representations and the textual domain. For instance, while a model might recognize visual patterns associated with a "cat," mapping these directly to the word "cat" during generation without explicit semantic grounding remains a challenge. This work aims to bridge this gap through explicit semantic modeling.

Methodology

The proposed system enhances the standard CNN-RNN architecture with a novel auxiliary task: Temporal Concept Prediction.

Architecture of the FDU video captioning system featuring a Temporal Concept Prediction module.

Visual Representation

The system utilizes an Inception-Resnet-V2 network, pre-trained on ImageNet, to extract 1,536-dimensional feature vectors for each video frame.

Concept Prediction

To close the visual-text domain gap, frame features are processed through a bi-directional LSTM (Bi-LSTM). A subsequent fully connected layer predicts specific visual concepts (e.g., "cat", "vegetable", "bowl") present in the video. These concept labels are derived directly from ground-truth sentence annotations. The hidden states generated during this process serve as "latent semantic representations."

Sentence Generation

The latent semantic representations are concatenated with the original visual features. A standard LSTM, enhanced with a dynamic temporal attention mechanism, uses this enriched feature set to generate the final caption word-by-word. The attention mechanism allows the model to dynamically focus on the most relevant frames at each time step of the generation process.

Experimental Setup

The model was trained and evaluated using several key datasets:

  • Training: The TGIF dataset, containing 100,000 videos/animated GIFs and 120,000 sentence annotations.
  • Validation: The TRECVID 2017 test set.
  • Testing: The TRECVID 2019 VTT dataset, consisting of 1,010 videos from Flickr and 1,044 videos from Vine.
  • Pre-training: ImageNet-1M was used for the initial training of the Inception-Resnet-V2 feature extractor.

Results and Discussion

The primary insight of this work is that explicit semantic grounding significantly aids the captioning process. Forcing the network to predict concepts as an auxiliary task creates a richer latent representation that bridges the gap between raw pixels and language.

Ablation study on the TRECVID 2017 validation set demonstrating the impact of concept prediction.

Ablation Study

Evaluation on the TRECVID 2017 validation set confirms the effectiveness of the concept prediction module. Performance improved across all metrics when the module was included: BLEU@4 rose from 7.56 to 8.04, METEOR from 12.75 to 13.23, and CIDEr from 31.12 to 32.00.

TRECVID 2019 Performance

On the unseen TRECVID 2019 test set, the final system achieved a BLEU@4 of 2.7, a METEOR score of 24.4, and a CIDEr score of 42.8. With only a single submitted run, this architecture secured a 3rd place overall ranking in the TRECVID 2019 VTT challenge.

Conclusion

The FDU system demonstrates that simple fusion of latent semantic features with standard visual features provides a strong signal for attention-based video captioning. While the authors only submitted a single run, the results suggest that further improvements could be achieved through standard competition techniques such as model ensembling or more extensive hyperparameter optimization.

Summary updated

BibTeX
@inproceedings{chen2019fdu,
  author = {Shaoxiang Chen and Yu-Gang Jiang},
  title = {{FDU Participation in TRECVID 2019 VTT Task}},
  booktitle = {TRECVID},
  year = {2019},
}
Figure