Utilizing Longer Context than Speech Bubbles in Automated Manga Translation

Hiroto Kaino, Soichiro Sugihara, Tomoyuki Kajiwara, Takashi Ninomiya, Joshua B. Tanner, Shonosuke IshiwatariCOLING 2024

TL;DR

This paper tackles the challenge of translating manga by proposing novel methods that leverage longer contextual information beyond individual speech bubbles. By incorporating previous scenes and bibliographic attributes of the manga, they achieve state-of-the-art translation quality, demonstrating the critical role of broader context in understanding and translating manga.

What Problem Does This Paper Address?

Manga, with its unique visual and textual structure, presents a significant challenge for automated machine translation (MT). Traditional MT approaches often treat text segments independently. In manga, this translates to translating each speech bubble in isolation. However, speech bubbles frequently contain incomplete sentences, rely on visual cues, or continue dialogues from previous bubbles or even scenes. This lack of context leads to inaccurate and unnatural translations. The paper highlights the rampant issue of unofficial pirated manga translations circulating globally, causing financial damage to publishers, and emphasizes the need for efficient and accurate automated translation to provide timely official versions. Existing methods, like "2+2 NMT" (using the previous bubble) and basic scene-based translation, improve upon isolated bubble translation but still struggle with context spanning across multiple scenes or incorporating broader manga-specific information.

Methods

The authors propose two novel methods to capture richer contextual information for manga MT:

  • Scene-based context-aware NMT: This method extends scene-based translation by incorporating the previous scene as context. It concatenates the text from the current scene with the text from the preceding scene, using special tokens to delineate scene and speech bubble boundaries. This allows the model to infer missing subjects or resolve ambiguities that are only clear when looking at the prior scene's dialogue.
  • Attribute-aware NMT: This approach leverages bibliographic attributes of the manga itself to inform the translation. It incorporates information such as the manga's series, author, publisher, magazine, and genre. By using a style control approach, the model can adapt its translation to the specific stylistic nuances or common expressions associated with these attributes, recognizing that different genres or authors might have distinct ways of phrasing things.

Datasets

The experiments were conducted using the Manga Corpus (Hinami et al., 2021). The authors randomly selected 50 series and 10 pages from the latest volume of each series to create validation and evaluation datasets. These datasets comprised 500 pages, 1,000 scenes, and 2,000 speech bubbles. The remaining pages were used for training.

Key Insights and Ideas

  • Context is King (Especially for Manga): The most striking insight is the significant improvement in translation quality when moving beyond single speech bubbles to consider broader contexts. Even just one previous scene provides substantial benefits.
  • The Power of Previous Scenes: The "Scene-based context-aware NMT" method demonstrates that looking at the immediately preceding scene is highly effective for resolving ambiguities and completing sentences.
  • Manga Attributes Matter for Style: The "Attribute-aware NMT" highlights that genre, author, and other metadata are not just metadata; they encode stylistic information that can significantly influence translation choices. For instance, the same sentence can be translated differently depending on whether the manga is a "Baseball" or "Robot" genre.
  • Zero-Anaphora Resolution is Crucial: The paper explicitly points out how longer context helps in resolving "zero-anaphora" – cases where subjects are omitted in Japanese but need to be supplied in English for a natural translation.

Key Results

  • The proposed "Scene-aware NMT" (combining scene-based context) achieved a 4.24 point improvement in BLEU score compared to the baseline, reaching state-of-the-art (SOTA) performance.
  • Both proposed methods (Scene-based context-aware NMT and Attribute-aware NMT) individually improved BLEU scores by over 2 points compared to the baseline.
  • The combination of both scene-based and attribute-aware methods yielded the best overall performance.
  • Analysis showed that considering one previous scene was the most effective for scene-based context, with performance deteriorating when considering more than two previous scenes.
  • Attribute-aware NMT showed improvements across most attributes, with the combination of all five attributes (series, author, publisher, magazine, genre) leading to the best results for attribute incorporation.

Limitations and Future Work

  • Limited Context Window: While the proposed methods extend context, the study found that considering more than one previous scene led to performance degradation. This suggests an optimal context window size needs further investigation.
  • Publisher Information: The paper notes that adding publisher information alone did not improve BLEU scores, possibly because publishers handle diverse manga, making it difficult to infer a consistent style from this attribute alone.
  • Future Directions: The authors suggest exploring multimodal MT (integrating visual information) and multilingual MT models pre-trained on large corpora to further enhance translation quality.

Summary updated

BibTeX
@inproceedings{kaino2024utilizing,
  author = {Hiroto Kaino and Soichiro Sugihara and Tomoyuki Kajiwara and Takashi Ninomiya and Joshua B. Tanner and Shonosuke Ishiwatari},
  title = {{Utilizing Longer Context than Speech Bubbles in Automated Manga Translation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure