English-to-Japanese Multimodal Machine Translation Based on Image-Text Matching of Lecture Videos

Ayu Teramen, Takumi Ohtsuka, Risa Kondo, Tomoyuki Kajiwara, Takashi NinomiyaACL 2024

TL;DR

This paper tackles the challenge of improving English-to-Japanese machine translation for lecture videos by leveraging visual information. They introduce a novel method for selecting the most relevant image from a set of three associated with each subtitle, significantly boosting translation quality compared to text-only or random image approaches.

What Problem Does This Paper Address?

Machine translation (MT) has made great strides, but translating spoken content, especially from lectures, remains challenging. Lecture videos offer a rich multimodal experience, combining audio, visuals (slides, speaker), and text (transcripts). While multimodal MT aims to leverage these different modalities, simply incorporating all available information isn't always beneficial. Specifically, for lecture videos, presentation slides often contain crucial information that complements the spoken word, but images of the speaker or audience might not directly aid translation. The problem is how to effectively select which visual information is most useful for improving translation quality, particularly for English-to-Japanese translation of lecture subtitles, where existing multimodal datasets are limited.

Methods

The core of this work lies in their proposed method for selecting the most relevant image to enhance machine translation. They leverage the BLIP (Bootstrapping Language-Image Pre-training) model, a powerful multimodal model, in two ways:

  • Embedding-based method: Both the English text (subtitle) and each of the three candidate images are encoded into embeddings using BLIP. The image with the highest cosine similarity to the text embedding is selected.
  • Captioning-based method: BLIP is used to generate an English caption for each image. The image whose generated caption has the highest BLEU score when compared to the input English text is selected.

These selected images are then fed into a multimodal MT model, specifically a Selective Attention model based on a Transformer architecture, to generate Japanese subtitles.

Datasets

The authors introduce the TAIL (English-to-Japanese Translation Corpus with Audio and Images from Lecture Videos) corpus. This corpus is constructed from TED lecture videos and includes:

  • Audio in English
  • Automatic Speech Recognition (ASR) sentences in English
  • Manually transcribed sentences in English
  • Reference translations in Japanese
  • Five sets of images per sentence, with three images specifically selected to correspond to the beginning, middle, and end of each sentence's timestamp.

The TAIL corpus is derived from the IWSLT2017 English-Japanese corpus, with added audio and image annotations. For their experiments, they used 70,000 sentence pairs for training, 2,669 for validation, and 2,371 for evaluation.

Key Insights and Ideas

  • Not all images are created equal: The paper strongly demonstrates that simply including images associated with a subtitle doesn't guarantee improved translation. The relevance of the image to the text is paramount.
  • Presentation slides are key: While not explicitly stated as a finding, the success of their method implies that images containing presentation material are more beneficial than those showing only the speaker.
  • BLIP's versatility: The use of BLIP for both embedding and captioning highlights its effectiveness in understanding the semantic relationship between images and text.
  • Image selection is crucial for multimodal MT: The performance gains are directly linked to the quality of the image selection process, suggesting that this is a critical bottleneck in image-guided MT.

Key Results

The experimental results, summarized in Table 1, show significant improvements:

  • ASR Text Translation: The embedding-based image selection method achieved a BLEU score of 9.48, a substantial improvement over the text-only baseline (3.94) and the random image baseline (8.98).
  • Transcribed Text Translation: Similarly, the embedding-based method achieved a BLEU score of 7.30 for transcribed text, outperforming the text-only baseline (4.73) and the random image baseline (8.97).
  • Human Evaluation: Human evaluation of image selection accuracy for the embedding-based method reached 0.785, significantly higher than the captioning-based method (0.410), indicating its superior ability to pick relevant images.

Limitations and Future Work

  • Limitations: The paper notes that some images derived from lecture videos might not be directly related to the subtitle text (e.g., images of the speaker). While their method aims to mitigate this, it's an inherent challenge with lecture video data. The captioning-based method's performance was lower than the embedding-based one, suggesting room for improvement in caption generation or its use for relevance scoring.
  • Future Work: The authors suggest further improving translation quality by combining multiple images, rather than relying on a single selected image. This could involve exploring more sophisticated fusion techniques or attention mechanisms that consider multiple visual inputs.

Summary updated

BibTeX
@inproceedings{teramen2024english,
  author = {Ayu Teramen and Takumi Ohtsuka and Risa Kondo and Tomoyuki Kajiwara and Takashi Ninomiya},
  title = {{English-to-Japanese Multimodal Machine Translation Based on Image-Text Matching of Lecture Videos}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure