Matesub: The Translated Subtitling Tool at the IWSLT2023 Subtitling Task

Simone PeroneACL 2023

TL;DR

This paper introduces Matesub, a web-based subtitling tool that leverages a cascade of neural models for automatic speech recognition (ASR), text segmentation, and machine translation (MT). Matesub aims to streamline the creation of subtitles for audio-visual content, offering a WYSIWYG editor for professional refinement and supporting a wide range of languages.

What Problem Does This Paper Address?

Creating high-quality subtitles for audio-visual content is a labor-intensive process. While automatic speech recognition (ASR) and machine translation (MT) technologies have advanced significantly, integrating them into a seamless workflow that produces professional-grade subtitles remains a challenge. Specifically, accurately segmenting speech into meaningful caption blocks, translating these segments, and ensuring they meet readability standards (like reading speed) requires a sophisticated pipeline. This paper addresses the need for an integrated tool that combines these advanced AI capabilities with a user-friendly interface for professional editors, enabling efficient and accurate subtitle generation across many languages.

Methods

Matesub employs a two-stage pipeline:

  1. Automatic Speech Recognition (ASR): This stage transcribes the audio content. Matesub can utilize either an in-house ASR model or a third-party commercial service. The output is expected in the Conversation Time Mark (CTM) format, which includes word hypotheses, start and end times, and an end-of-sentence flag.
  2. Captions & Subtitles Service: This module takes the ASR output and generates subtitle files. It comprises two main neural components:
  3. Text Segmenter: This component, based on sequence-to-sequence models, segments the transcribed text into caption blocks and lines. It uses special symbols (<eob> for end-of-block, <eol> for end-of-line) to mark boundaries, trained on pairs of unsegmented and segmented text.
  4. Neural Machine Translation (NMT): Matesub integrates with ModernMT via a REST API for translation. The NMT system uses the Transformer architecture and is trained on a vast amount of data, covering many language pairs. It provides multiple translation hypotheses (up to 16), from which the best one is selected based on reading speed constraints and confidence.

The pipeline also includes modules for generating SRT files and merging them into a single JSON file, along with a WYSIWYG editor that allows users to review and correct both translation and segmentation errors.

Datasets

The paper mentions that the Matesub pipeline's neural models (ASR, text segmenter, and MT) were trained on "more resources than those allowed for the constrained condition." However, specific details about the training datasets used for Matesub itself are not provided. The evaluation was performed on the IWSLT 2023 Subtitling shared task test set.

Key Insights and Ideas

  • Cascade Architecture for Subtitling: The paper demonstrates the effectiveness of a cascaded approach, combining ASR, segmentation, and MT, to build a comprehensive subtitling tool.
  • Importance of Segmentation: Accurate text segmentation is crucial for generating natural-sounding subtitles, and Matesub's segmenter, trained on specific boundary markers, plays a key role.
  • NMT for Translation Quality and Speed: Leveraging a powerful NMT system like ModernMT, with its ability to provide multiple translations and consider reading speed, is vital for producing high-quality, compliant subtitles.
  • WYSIWYG Editor for Professional Refinement: The inclusion of a user-friendly WYSIWYG editor is essential for professional subtitlers to easily correct and refine the automatically generated output, bridging the gap between AI capabilities and human expertise.
  • Language Agnosticism: The system is designed to support "any pair from about 60 languages and their variants," highlighting its broad applicability.

Key Results

The paper reports the results of Matesub's submission to the IWSLT 2023 Subtitling shared task. The submission was labeled "unconstrained" as no specific adaptations were made to the general-purpose pipeline. The scores obtained on the test set were in line with those achieved on development sets.

Key findings from the evaluation include:

  • The TED domain was the easiest for pure speech translation.
  • The EPTV domain presented challenges comparable to ITV and PELOTON for German target language.
  • Metrics like BLEURT and ChrF suggested high quality for TED and Spanish EPTV subtitles, while ITV, PELOTON, and German EPTV required more post-editing.
  • The compliance of subtitles (measured by CPS, CPL, LPB) was generally high, with CPS (characters per second) being under the 21 CPS threshold in about 85% of subtitles, averaging around 18.5 CPS.
  • The CPL (caption compliance) and LPB (line break compliance) metrics were reported as "almost perfect."

Limitations and Future Work

  • Block-wise Translation: The translation of each block is performed independently, without considering surrounding blocks. While this simplifies the process and aids in meeting reading speed requirements, it can slightly degrade translation quality.
  • Limited Context in Translation: The authors note that the translation of each block is done "without looking at the context, i.e. at the surrounding blocks." This is a trade-off for facilitating reading speed requirements.
  • No Hard Rules: Matesub relies solely on machine learning methods and does not implement explicit hard rules, which might be a limitation in certain scenarios.

Future work is not explicitly detailed, but the success in the IWSLT 2023 task and the reported high compliance metrics suggest that further refinement of the NMT and segmentation models, potentially incorporating cross-block context, could lead to even higher quality subtitles.

Summary updated

BibTeX
@inproceedings{perone2023matesub,
  author = {Simone Perone},
  title = {{Matesub: The Translated Subtitling Tool at the IWSLT2023 Subtitling Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure