Abstract
Manga, or comics, which are a type of multimodal artwork, have been left behind in the recent trend of deep learning applications because of the lack of a proper dataset. Hence, we built Manga109, a dataset consisting of a variety of 109 Japanese comic books (94 authors and 21,142 pages) and made it publicly available by obtaining author permissions for academic use. We carefully annotated the frames, speech texts, character faces, and character bodies; the total number of annotations exceeds 500k. This dataset provides numerous manga images and annotations, which will be beneficial for use in machine learning algorithms and their evaluation. In addition to academic use, we obtained further permission for a subset of the dataset for industrial use. In this article, we describe the details of the dataset and present a few examples of multimedia processing applications (detection, retrieval, and generation) that apply existing deep learning methods and are made possible by the dataset.
Introduction
While deep learning has revolutionized natural image processing, manga has largely been left behind due to data scarcity and legal hurdles. Most existing comic datasets are either too small for robust training or consist of low-quality, public-domain scans. Furthermore, manga is a commercial product protected by strict copyright, making it difficult for researchers to share data or publish images without navigating complex permission webs.
Manga is a unique multimodal medium where layout (frames), text (speech bubbles), and stylized art (character faces/bodies) interact to tell a story. The "Manga109" project addresses these challenges by introducing the first large-scale, professionally authored, and fully annotated dataset for Japanese comics. By securing legal permissions for 109 volumes and providing over 500,000 manual annotations, the researchers have created a vital sandbox for training deep learning models in manga-specific computer vision tasks.
Methodology and Annotation Pipeline
The researchers spent years building Manga109, focusing on a rigorous manual annotation process. They collaborated with professional author Ken Akamatsu and the J-Comic Terrace Corporation to select 109 titles from the "Manga Library Z" archive, covering a wide range of genres and eras (1970s–2010s).
A key preprocessing step involved creating "double-sided pages." Since manga artists often draw high-impact scenes across two pages, the researchers concatenated left and right pages into single 1654 × 1170 pixel images to ensure annotations didn't cut characters in half.
The annotation pipeline involved 72 workers over 12 months, followed by a 10-month quality control phase. They developed a custom web-based tool to define four primary object categories: Frames, Text, Faces, and Bodies, while also linking character names to those boxes.
Dataset Characteristics
The primary Manga109 dataset contains 109 volumes, 21,142 pages, and 94 authors. It includes 103k frame boxes, 147k text boxes, 118k face boxes, and 157k body boxes. A subset, Manga109-s (87 volumes), is specifically cleared for industrial and commercial use.
Key insights from the dataset creation include:
- Professional Quality: Models trained on professional manga ensure the learning of actual industry stylistic conventions.
- Scale Effect: Increasing the training set from 10 to 99 volumes significantly boosted text detection performance (from ~84% to ~91% AP).
- Multimodal Synergy: Annotating both text and character names enables advanced tasks like "speaker detection."
Multimedia Applications and Results
The utility of Manga109 was demonstrated through three proof-of-concept applications:
- Text Detection: Using a Single Shot Multibox Detector (SSD512), the researchers achieved 91.8 Average Precision (AP).
- Sketch-Based Retrieval: Deep features from AlexNet proved superior to traditional methods for finding characters based on hand-drawn sketches.
- Character Generation: Using Progressive Growing of GANs (PGGAN), the dataset's 118k face annotations were sufficient to generate high-quality, diverse manga character faces.
Conclusion and Future Work
While Manga109 provides a massive leap forward, future work could include instance segmentation for more precise outlines or detailed labels for onomatopoeia. The current manual process was labor-intensive, suggesting that future iterations might use the existing dataset to "pre-annotate" new content for human correction. Despite copyright constraints, Manga109 stands as a foundational resource for the intersection of AI and comic art.
Summary updated