Keyframe Segmentation and Positional Encoding for Video-guided Machine Translation Challenge 2020

Tosho Hirasawa, Zhishen Yang, Mamoru Komachi, Naoaki OkazakiarXiv 2020 · OpenReview

Abstract

Video-guided machine translation as one of multimodal neural machine translation tasks targeting on generating high-quality text translation by tangibly engaging both video and text. In this work, we presented our video-guided machine translation system in approaching the Video-guided Machine Translation Challenge 2020. This system employs keyframe-based video feature extractions along with the video feature positional encoding. In the evaluation phase, our system scored 36.60 corpus-level BLEU-4 and achieved the 1st place on the Video-guided Machine Translation Challenge 2020.

Introduction

Multimodal Machine Translation (MMT) aims to improve translation quality by providing the model with non-linguistic context, such as images. Video-guided Machine Translation (VMT) takes this a step further by using video data. However, the challenge with video is twofold: capturing temporal information and ensuring feature quality. Standard models often extract features from frames but discard the information about when those frames occurred, treating the video like a "bag of images." Furthermore, videos contain significant redundancy, making it critical to identify keyframes that contain the most relevant information for describing objects, scenes, and actions.

Methodology

The proposed system is built on a modified hierarchical attention model, utilizing two separate attention mechanisms to focus on the source text and video representations respectively.

Keyframe-based Visual Feature Extraction

Instead of processing every frame, the system uses keyframes to segment the video and extract high-quality visual "summaries." Three types of features are utilized:

  • Action Features: Extracted using a non-local neural network (ResNet-101 backbone) pre-trained on the Kinetics-400 dataset.
  • Object Features: Extracted using ResNet-152 (trained on ImageNet).
  • Scene Features: Extracted using ResNet-50 (trained on Place365).
Example of a keyframe capturing a specific action in a video sequence.

Positional Encoding

To address the loss of temporal information, the researchers added sinusoidal positional encodings to the video features. This technique, borrowed from the Transformer architecture, "injects" the chronological order of the video segments into the model, allowing the attention mechanism to distinguish between the beginning, middle, and end of an action.

Results and Discussion

The system was evaluated on the Video-guided Machine Translation Challenge 2020 dataset. The results, summarized in the table below, demonstrate that temporal order is essential; adding video features without positional encoding actually performed worse than a text-only baseline.

Performance comparison of various model configurations on the validation and public test sets.

The final ensemble model, which combined action, object, and scene features, achieved a BLEU-4 score of 36.60 on the public test set, securing 1st place in the challenge. Qualitative improvements were also noted, with the ensemble model proving better at identifying specific subjects (e.g., "swimming athletes") and complex actions (e.g., "practicing underwater dance tricks") compared to baseline versions.

Conclusion

The success of this approach highlights that temporal awareness and visual diversity are critical for high-quality video-guided translation. Future work will focus on finding more integrated ways to combine diverse visual cues within a single model to reduce reliance on heavy ensemble methods.

Summary updated

Abstract

Video-guided machine translation as one of multimodal neural machine translation tasks targeting on generating high-quality text translation by tangibly engaging both video and text. In this work, we presented our video-guided machine translation system in approaching the Video-guided Machine Translation Challenge 2020. This system employs keyframe-based video feature extractions along with the video feature positional encoding. In the evaluation phase, our system scored 36.60 corpus-level BLEU-4 and achieved the 1st place on the Video-guided Machine Translation Challenge 2020.
BibTeX
@misc{hirasawa2020keyframe,
  author = {Tosho Hirasawa and Zhishen Yang and Mamoru Komachi and Naoaki Okazaki},
  title = {{Keyframe Segmentation and Positional Encoding for Video-guided Machine Translation Challenge 2020}},
  year = {2020},
  eprint = {2006.12799},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.12799},
}
Figure