TL;DR
This paper introduces a novel self-supervised learning pipeline for robust face and body detection in drawings, a challenging domain due to vast stylistic variations. By combining style-transferred pre-training with a modified teacher-student architecture and OHEM loss, the approach achieves state-of-the-art performance with significantly reduced annotation effort.
What Problem Does This Paper Address?
Detecting objects like faces and bodies in drawings, particularly in styles like cartoons and comics, is significantly harder than in real-world images. This is primarily due to the immense diversity in artistic styles, character designs, and drawing techniques, which means a detector trained on one style often performs poorly on another. Traditional supervised learning methods require extensive manual annotation for each new style, which is prohibitively expensive and time-consuming. While self-supervised learning has shown promise in other domains, adapting it effectively to the unique challenges of artistic drawings, especially with limited labeled data, remains an open problem.
Methods
The authors propose a three-stage pipeline:
- Stage 1: Style Transferred Pre-Training: This stage leverages large, labeled real-world image datasets (like COCO and WIDER FACE) and applies various style transfer techniques to generate cartoonized versions. This creates a diverse synthetic dataset that exposes the model to a range of artistic styles before it ever sees actual drawings. They experiment with different combinations of styles to find the optimal mix for pre-training.
- Stage 2: Self-Supervised Pre-Training: Here, a teacher-student network architecture is employed. The teacher network processes an unaugmented image and generates "pseudo-ground truth" labels. The student network, processing a heavily augmented version of the same image, is trained to predict these labels. A key innovation is a modified OHEM (Online Hard Example Mining) loss function, which helps mitigate false positives and negatives by carefully selecting proposals for loss calculation. They also introduce a periodic weight update mechanism for the teacher network to stabilize training.
- Stage 3: Fine-Tuning: Finally, the pre-trained model from Stage 2 is fine-tuned on a small amount of labeled drawing data. This stage adapts the general drawing detector to specific target domains or datasets.
Datasets
- Real-world Image Datasets: COCO, WIDER FACE (used for style transfer and initial pre-training).
- Drawing Datasets (for training and evaluation):
- iCartoonFace (iCF)
- Manga 109
- DCM 772
- Comic2k (C2k)
- Watercolor2k (W2k)
- Clipart1k (Clk)
- COMICS (crawled, unlabeled)
The paper notes that while many drawings are available online, labeled datasets are scarce and often cover only specific sub-domains.
Key Insights and Ideas
- Style Transfer is Crucial for Generalization: Applying a diverse set of style transfer techniques to real-world images before training on drawings significantly boosts performance. This forces the model to learn shape and structure over specific textures, making it more robust to unseen artistic styles.
- "All Styles" Outperforms Specific Styles: Combining a wide range of styles during pre-training yielded better results than using only a few "best" styles or a single style. This suggests that maximum diversity is key for broad domain adaptation.
- Modified OHEM Loss for Self-Supervision: The proposed OHEM loss, with custom confidence thresholds for positive and negative examples, is more effective than standard losses (like SimOTA) for self-supervised learning in drawings, helping to filter out noisy pseudo-labels.
- Teacher-Student Weight Synchronization: Periodically synchronizing the teacher's weights with the student's (rather than just backpropagating through the student) helps stabilize training and prevent performance degradation, especially when dealing with domain shifts.
- SGD is Preferred Over Momentum Optimizers: For the teacher-student weight update, standard SGD performed better than optimizers with momentum, likely due to the noisy nature of pseudo-ground truth labels.
Key Results
- The proposed method achieves state-of-the-art (SOTA) performance on most drawing datasets when fine-tuned with limited labeled data.
- Compared to models trained without style transfer pre-training, the proposed method shows an improvement of ~7% AP.
- Even with very few labeled images (e.g., 64 or 512), the approach provides an up to ~24% performance increase compared to random initialization.
- The model achieves close scores to ACFD [Zhang et al., 2020a] and outperforms other SOTA models, demonstrating its effectiveness.
Limitations and Future Work
- Domain-Specific Fine-Tuning: While the pre-training is domain-adaptive, the authors still fine-tune the model on individual datasets to achieve optimal performance, acknowledging that each drawing dataset has its own stylistic distribution.
- Limited Annotation for Specific Styles: The paper highlights the ongoing challenge of comprehensive stylistic coverage in existing labeled drawing datasets, particularly for contemporary US and Western comics.
- Future Work: The authors suggest that their pipeline is not tied to a specific baseline detector and can be applied to other architectures. Further exploration of more advanced style transfer techniques and self-supervised learning strategies could yield even better results.
Summary updated