Video-guided Machine Translation with Spatial Hierarchical Attention Network

Weiqi Gu, Haiyue Song, Chenhui Chu, Sadao KurohashiACL 2021

TL;DR

This paper tackles the challenge of word sense ambiguity in video-guided machine translation (VMT) by proposing a novel system that leverages both spatial and motion features from videos. Their Spatial Hierarchical Attention Network (Spatial HAN) effectively captures contextual information from objects within frames and across frames, leading to improved translation quality, particularly for noun sense ambiguity.

What Problem Does This Paper Address?

Machine translation, while advanced for text-only domains, struggles with ambiguity in real-time or multimodal contexts like spoken language or sports commentary. Specifically, word sense ambiguity (where a word can have multiple meanings) for both verbs and nouns can significantly degrade translation quality. While previous multimodal machine translation (MMT) approaches have used visual data to help, they often focus on image-based translation or only utilize motion features for verb disambiguation, leaving noun ambiguity largely unaddressed. This paper aims to solve both verb and noun sense ambiguity by more effectively utilizing the rich information present in videos.

Methods

The proposed system builds upon a Video-Guided Machine Translation (VMT) baseline model. The core innovation is the introduction of a Spatial Hierarchical Attention Network (Spatial HAN).

  • VMT Baseline: This baseline uses a text encoder (Bi-GRU) for source sentences and a motion encoder (Bi-GRU) on pre-extracted motion features (from I3D models) to generate text and motion representations. These are then fed into a target decoder with attention mechanisms to produce the translation.
  • Spatial HAN: This module is designed to extract richer spatial information.
  • It first extracts object-level spatial features from each frame using Faster R-CNN.
  • A hierarchical attention mechanism is then applied:
  • An object-level attention layer summarizes information from all detected objects within a frame.
  • A frame-level attention layer then summarizes representations from all ordered frames to create a video-level abstraction.
  • Positional encoding is applied to both motion and spatial features to preserve their order.
  • Target Decoder: The final decoder integrates text, motion, and the new spatial representations from Spatial HAN through an attention mechanism to generate the translated output.

Datasets

  • VATEX: This is the primary dataset used for training and evaluation. It's a large-scale dataset for video-and-language research, containing video clips with parallel English-Chinese descriptions. The authors note that for their spatial feature extraction, they had to re-collect data, and a small percentage of clips were unavailable, making the comparison slightly unfair.

Key Insights and Ideas

  • Spatial information is crucial for noun disambiguation: While motion features help with verb ambiguity, the paper strongly suggests that detailed spatial information, especially at the object level and across frames, is key to resolving noun sense ambiguity.
  • Hierarchical attention effectively models spatial context: The proposed Spatial HAN's ability to first process objects within frames and then aggregate frame-level information into a video-level representation is a powerful way to capture complex visual context.
  • Videos offer more than just motion: The paper highlights that the sequential nature of video frames provides not only motion cues but also rich spatial context that can be exploited for translation.
  • Alignment is challenging but important: The authors observe that comparable results are achieved with and without motion features, suggesting potential misalignment issues between motion, spatial, and text features. This points to future work in better aligning these modalities.

Key Results

  • The proposed VMT system with Spatial HAN achieved a corpus-level BLEU-4 score of 35.86 on the VATEX test set.
  • This represents an improvement of 0.51 BLEU-4 score over the state-of-the-art VMT baseline model (Hirasawa et al., 2020).
  • An ablation study showed that the Spatial HAN module itself is effective, and the middle layer within the HAN was found to be dispensable, likely due to the positional encoding already providing contextual information.
  • Qualitative examples (Figure 4) demonstrate the model's ability to correctly translate noun phrases that were ambiguous for the baseline model, and to avoid misinterpreting verbs as nouns.

Limitations and Future Work

  • Data availability: The authors acknowledge that a small percentage of video clips were unavailable for spatial feature extraction, making the comparison slightly unfair.
  • Alignment: The paper suggests that the performance could be further improved by better aligning text, motion, and spatial features. This is identified as a key direction for future research.
  • Human evaluation: While quantitative results are strong, the paper mentions conducting a pairwise human evaluation which showed the model's superiority in noun sense disambiguation, but further details are not elaborated.

Summary updated

BibTeX
@inproceedings{gu2021video,
  author = {Weiqi Gu and Haiyue Song and Chenhui Chu and Sadao Kurohashi},
  title = {{Video-guided Machine Translation with Spatial Hierarchical Attention Network}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure