Introduction
Translating manga presents unique challenges for standard Machine Translation (MT) systems. Unlike traditional prose, manga is inherently multimodal; the meaning of a sentence often depends on visual cues such as the speaker's identity, gender, surroundings, or even events from previous pages. Text-only models frequently struggle with linguistic ambiguities, contextual omissions (common in Japanese), and fragmented text split across multiple speech bubbles. Consequently, many manga titles remain untranslated due to the high cost of manual localization. This research explores whether Multimodal Large Language Models (MLLMs), such as GPT-4 Turbo, can leverage visual information to resolve these ambiguities and improve translation quality.
Methodology
The researchers evaluated several strategies to determine the optimal level of context for manga translation:
- Granularity Levels: Comparison between Line-by-Line (LBL) and Page-by-Page (PBP) translation.
- Multimodal Integration (PBP-VIS): Providing the model with both the text and the raw image of the manga page.
- Spatial Awareness (PBP-VIS-NUM): To help the model map text to specific panels, a "Numbered Bubble" approach was tested, where text in the image is replaced by numbers corresponding to a provided text list.
- Long-Context Strategies: Experimentation with context windows beyond a single page, including a 3-page window and a "Chain of Density" (VBP-VIS-COD) approach that provides a rolling, entity-dense summary of the narrative.
Datasets
The study utilized two primary benchmarks:
- OpenMantra (JA-EN): A small-scale benchmark of five manga volumes (approx. 214 pages).
- JA-PL Dataset: A new, large-scale parallel Japanese-Polish dataset introduced by the authors. It features 400 professionally translated pages from the series Love Hina, making it the largest publicly available manga translation dataset for research.
The Role of Visual Context
A key finding of the study is that visual information acts as a critical disambiguator. For instance, the Japanese word "Suzume" can mean "sparrow" or be a character's name. Without visual context, models often default to the literal translation. By "seeing" the character on the page, the MLLM can correctly identify the proper noun.
Ablation studies further confirmed this dependency; when key visual regions (e.g., a TV being turned off) were obscured, translation accuracy dropped significantly, proving that the model actively utilizes visual cues rather than relying solely on text.
Results and Evaluation
The proposed multimodal methods achieved state-of-the-art (SOTA) results on the JA-EN OpenMantra dataset, outperforming specialized manga-MT models and Google Translate. On the JA-PL dataset, the PBP-VIS model achieved a ChrF score of 25.6, nearly doubling the performance of the Google Translate baseline (22.3).
Human evaluation using Multidimensional Quality Metrics (MQM) showed that PBP-VIS significantly outperformed generic MT baselines. However, a gap remains between MLLM outputs and professional human translations, particularly regarding "critical" errors that can disrupt narrative immersion.
Conclusion
This research demonstrates that incorporating visual context is essential for high-quality manga translation. While page-level multimodal context provides a "Goldilocks" zone of performance, challenges remain in maintaining narrative consistency over long arcs and reducing the frequency of critical errors. Future work will focus on developing more efficient memory mechanisms and reducing dependency on expensive, closed-source commercial models.
Summary updated