Context-Informed Machine Translation of Manga using Multimodal Large Language Models

Philip Lippmann, Konrad Skublicki, Joshua B. Tanner, Shonosuke Ishiwatari, Jie Yang

Introduction

Translating manga presents unique challenges for standard Machine Translation (MT) systems. Unlike traditional prose, manga is inherently multimodal; the meaning of a sentence often depends on visual cues such as the speaker's identity, gender, surroundings, or even events from previous pages. Text-only models frequently struggle with linguistic ambiguities, contextual omissions (common in Japanese), and fragmented text split across multiple speech bubbles. Consequently, many manga titles remain untranslated due to the high cost of manual localization. This research explores whether Multimodal Large Language Models (MLLMs), such as GPT-4 Turbo, can leverage visual information to resolve these ambiguities and improve translation quality.

Methodology

The researchers evaluated several strategies to determine the optimal level of context for manga translation:

  1. Granularity Levels: Comparison between Line-by-Line (LBL) and Page-by-Page (PBP) translation.
  2. Multimodal Integration (PBP-VIS): Providing the model with both the text and the raw image of the manga page.
  3. Spatial Awareness (PBP-VIS-NUM): To help the model map text to specific panels, a "Numbered Bubble" approach was tested, where text in the image is replaced by numbers corresponding to a provided text list.
  4. Long-Context Strategies: Experimentation with context windows beyond a single page, including a 3-page window and a "Chain of Density" (VBP-VIS-COD) approach that provides a rolling, entity-dense summary of the narrative.
The PBP-VIS-NUM approach uses numbered bubbles to help the model associate text with specific spatial locations on the manga page.

Datasets

The study utilized two primary benchmarks:

  • OpenMantra (JA-EN): A small-scale benchmark of five manga volumes (approx. 214 pages).
  • JA-PL Dataset: A new, large-scale parallel Japanese-Polish dataset introduced by the authors. It features 400 professionally translated pages from the series Love Hina, making it the largest publicly available manga translation dataset for research.

The Role of Visual Context

A key finding of the study is that visual information acts as a critical disambiguator. For instance, the Japanese word "Suzume" can mean "sparrow" or be a character's name. Without visual context, models often default to the literal translation. By "seeing" the character on the page, the MLLM can correctly identify the proper noun.

Comparison of translation methods: Visual context allows the PBP-VIS model to correctly identify "Suzume-chan" as a name rather than translating it as "Little Sparrow."

Ablation studies further confirmed this dependency; when key visual regions (e.g., a TV being turned off) were obscured, translation accuracy dropped significantly, proving that the model actively utilizes visual cues rather than relying solely on text.

Results and Evaluation

The proposed multimodal methods achieved state-of-the-art (SOTA) results on the JA-EN OpenMantra dataset, outperforming specialized manga-MT models and Google Translate. On the JA-PL dataset, the PBP-VIS model achieved a ChrF score of 25.6, nearly doubling the performance of the Google Translate baseline (22.3).

Human evaluation using Multidimensional Quality Metrics (MQM) showed that PBP-VIS significantly outperformed generic MT baselines. However, a gap remains between MLLM outputs and professional human translations, particularly regarding "critical" errors that can disrupt narrative immersion.

Conclusion

This research demonstrates that incorporating visual context is essential for high-quality manga translation. While page-level multimodal context provides a "Goldilocks" zone of performance, challenges remain in maintaining narrative consistency over long arcs and reducing the frequency of critical errors. Future work will focus on developing more efficient memory mechanisms and reducing dependency on expensive, closed-source commercial models.

Summary updated

BibTeX
@inproceedings{lippmann2025context,
  author = {Philip Lippmann and Konrad Skublicki and Joshua B. Tanner and Shonosuke Ishiwatari and Jie Yang},
  title = {{Context-Informed Machine Translation of Manga using Multimodal Large Language Models}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2025},
}
Figure