UTS ISA Submission at the TRECVID 2019 Video to Text Description Task

Qi Rao, Guang Li, Yi Yang, Feng Zhang, Ziwei WangTrecvid 2019

Introduction

Video captioning, the process of automatically generating natural language descriptions for video sequences, represents a significant intersection of computer vision and natural language processing. The task requires a model to not only identify objects and actions within a temporal sequence but also to synthesize this information into a coherent, grammatically correct sentence. This paper details the submission by the UTS ISA team for the TRECVID 2019 Video to Text (VTT) Description Task.

The TRECVID VTT task presents unique challenges due to the high variability in video content, ranging from short social media clips (Vine) to diverse Flickr videos. Furthermore, the evaluation of such tasks is complicated by the differing priorities of automated metrics like BLEU, METEOR, and CIDEr. The UTS ISA team addressed these challenges through a combination of state-of-the-art visual features, a robust encoder-decoder architecture, and a novel controllable beam search strategy.

System Architecture

The proposed framework follows a three-stage pipeline: feature extraction, sequence modeling, and ensemble decoding.

Visual Feature Extraction

To provide the model with a comprehensive understanding of the video content, the team utilized a multi-pronged feature extraction strategy. Spatial information, focusing on objects and scene context, was captured using ResNeXt-WSL and EfficientNet. To capture the temporal dynamics and specific actions occurring over time, the Kinect-i3d (Inception-v3 based 3D CNN) was employed. These features were concatenated to form a rich, high-dimensional representation of the video input.

Encoder-Decoder Framework

The core of the captioning system is a Recurrent Neural Network (RNN) based encoder-decoder.

  • Encoder: A bidirectional RNN processes the sequence of visual features. A random sampler is integrated into the encoder to improve the model's robustness to varying video lengths.
  • Decoder: A Gated Recurrent Unit (GRU) based language model serves as the decoder. It utilizes an attention mechanism to dynamically focus on specific parts of the encoded visual sequence during word generation.
  • Optimization: Several advanced training techniques were implemented to enhance performance, including embedding tying (sharing weights between input and output embeddings), weight-drop LSTM for regularization, label smoothing, and sequence-level optimization to directly target evaluation metrics.

Ensemble and Decoding Strategy

Recognizing that individual models may have specific biases, the team developed an expandable ensemble module. This module aggregates predictions from multiple models trained on different subsets of data or using different feature combinations.

A critical component of the decoding phase is the controllable beam search. This mechanism allows for the explicit adjustment of the generated sentence length, which is vital because different evaluation metrics respond differently to sentence length and complexity.

Datasets and Training

The models were trained on a diverse collection of six open-source datasets: MSVD, MSR-VTT, TGIF, and the VTT datasets from the 2016–2018 TRECVID cycles. These datasets were categorized into three distinct training sets based on their linguistic styles to improve the ensemble's breadth.

The evaluation was conducted on the TRECVID VTT 2019 test set, comprising approximately 2,000 videos sourced from Twitter Vine and Flickr Creative Commons. Each video in this set is accompanied by five human-written ground-truth annotations.

Discussion and Key Insights

The Metric Trade-off

A significant finding of this research is the inherent tension between different evaluation metrics. The authors observed that increasing the target sentence length in the controllable beam search led to improvements in CIDEr and METEOR scores. However, this same increase often resulted in a decrease in BLEU4 and CIDEr-D scores. This suggests that while longer sentences may capture more descriptive detail (rewarded by METEOR), they may also introduce more opportunities for precision errors (penalized by BLEU).

Model Diversity and Generalization

The ensemble strategy benefited greatly from model diversity. By training on different feature sets and data partitions, the team ensured that the constituent models made non-overlapping errors, which could be mitigated during the aggregation phase.

The team also explored the limits of model depth, testing RNN architectures with up to 512 layers. While these deep models showed superior performance on specific validation sets, they often failed to generalize to the broader TRECVID test set. Consequently, the final submission prioritized a balance between model depth and generalization capability.

Conclusion

The UTS ISA team's submission demonstrated the efficacy of combining high-level spatial and temporal features with a flexible ensemble framework. By targeting a balanced sentence length of approximately 20 words via controllable beam search, the team achieved a robust performance across multiple metrics. Future work will likely transition from RNN-based structures to Transformer architectures to better capture long-range dependencies and further improve the linguistic quality of generated captions.

Summary updated

BibTeX
@inproceedings{rao2019uts,
  author = {Qi Rao and Guang Li and Yi Yang and Feng Zhang and Ziwei Wang},
  title = {{UTS ISA Submission at the TRECVID 2019 Video to Text Description Task}},
  booktitle = {Trecvid},
  year = {2019},
}
Figure