Improving Object Detection in Art Images Using Only Style Transfer

David Kadish, Sebastian Risi, Anders Sundnes Løvlie

TL;DR

This paper tackles the "cross-depiction problem" where object detection models trained on photos struggle with artistic images. They introduce a novel method of generating a large, stylized training dataset (StyleCOCO) by applying artistic style transfer to existing photo datasets. This approach significantly improves object detection performance on art images, outperforming previous state-of-the-art methods.

What Problem Does This Paper Address?

Modern object detection systems, powered by deep neural networks, have achieved remarkable success on photographic datasets. However, they falter when presented with art images like paintings and drawings. This "cross-depiction problem" arises because these networks tend to rely heavily on texture rather than shape for identification, a bias that art often subverts. The lack of large, high-quality, labeled datasets for art images exacerbates this issue, hindering progress in applications like art historical research and digital humanities.

Methods

The core of this paper is the creation of a new dataset, StyleCOCO, and its application in training an object detector.

  • Style Transfer for Dataset Generation: The authors leverage AdaIN (Adaptive Instance Normalization) style transfer, a technique known for its ability to transfer the artistic style of one image to the content of another. They use the "Painter by Numbers" dataset for artistic styles and apply it to images from the COCO dataset that are already annotated for object detection (specifically, people). This process generates a large dataset of stylized images that retain the original bounding box annotations.
  • Object Detection Model: A Faster R-CNN object detection network with a ResNet-152 backbone, pre-trained on ImageNet, is used. This model is then fine-tuned on the newly created StyleCOCO dataset.
  • Training Strategy: The training involves fine-tuning the Faster R-CNN model on the StyleCOCO training and validation sets. They investigate the impact of the number of training images on performance and employ early stopping to prevent overfitting.

Datasets

  • COCO (2017): Used as the source of photographic images with existing object detection annotations.
  • Painter by Numbers: Used as the source of artistic styles for the style transfer process.
  • StyleCOCO: A newly generated dataset consisting of 58,672 stylized training images and 2,688 validation images. This dataset is derived from COCO images that contained people, with artistic styles applied. Images without people were removed to provide better negative examples.
  • People-Art: Used as the primary testing dataset for evaluating the performance of the trained object detector. This dataset consists of art images with labeled bounding boxes for people.

Key Insights and Ideas

  • Texture vs. Shape Bias: The paper reinforces the finding that neural networks are heavily biased towards texture, which is a major hurdle for recognizing objects in artistic representations that often prioritize shape and form.
  • Style Transfer as a Dataset Augmentation Tool: The core innovation is demonstrating that style transfer can be effectively used to create large-scale, annotated datasets for domains where such data is scarce, bridging the gap between photographic and artistic domains.
  • Generalization to Unseen Styles: The success of fine-tuning on a stylized dataset (StyleCOCO) to perform well on a distinct art dataset (People-Art) suggests that this approach can lead to models that generalize better to various artistic styles, even those not explicitly seen during training.
  • The Power of Scale: The experiments show a clear correlation between the number of stylized training images and improved detection performance, highlighting the importance of dataset size, even when using stylized data.

Key Results

  • Significant Performance Improvement: The proposed method achieved a state-of-the-art AP50 score of 0.68 on the People-Art dataset, a 10-percentage-point improvement over previous best results (0.58).
  • Impact of Training Data Size: Performance on the People-Art dataset improved significantly as the number of training images in StyleCOCO increased, particularly up to around 32,000 images. While AP50 showed a slight dip at the largest dataset size, AP and AP75 continued to rise.
  • Outperforming Human Annotators: In some instances, the trained network was able to correctly detect people in artwork that were not labeled in the original People-Art dataset, demonstrating its ability to sometimes surpass human annotation capabilities.

Limitations and Future Work

  • Bias in Style Sources: The authors acknowledge that the style transfer process relies on datasets like "Painter by Numbers" and WikiArt, which may have inherent biases (e.g., geographical or thematic skew). This could lead to models that perform better on certain types of art than others.
  • Failure Modes: The paper identifies failure modes including difficulty with highly abstract representations, complex visual scenes, and potential biases inherited from the ImageNet backbone and COCO dataset.
  • Future Work:
    • Targeted Style Transfer: Generating style transfer datasets using unlabelled images directly from the target art domain could lead to even more specialized and accurate models.
  • Expanding to Other Object Categories: The technique could be extended to train detectors for other object categories beyond people, leveraging the vastness of datasets like COCO.
  • Exploring Other Annotations: Investigating the transfer of segmentation and keypoint detection tasks to art images using style transfer.
  • Analyzing Bias: Further research into identifying and mitigating biases inherited from the backbone, source image datasets, and style transfer sources.

Summary updated

Abstract

Despite recent advances in object detection using deep learning neural networks, these neural networks still struggle to identify objects in art images such as paintings and drawings. This challenge is known as the cross depiction problem and it stems in part from the tendency of neural networks to prioritize identification of an object's texture over its shape. In this paper we propose and evaluate a process for training neural networks to localize objects — specifically people — in art images. We generate a large dataset for training and validation by modifying the images in the COCO dataset using AdaIn style transfer. This dataset is used to fine-tune a Faster R-CNN object detection network, which is then tested on the existing People-Art testing dataset. The result is a significant improvement on the state of the art and a new way forward for creating datasets to train neural networks to process art images.
BibTeX
@inproceedings{kadish2021improving,
  author = {David Kadish and Sebastian Risi and Anders Sundnes Løvlie},
  title = {{Improving Object Detection in Art Images Using Only Style Transfer}},
  booktitle = {IJCNN},
  year = {2021},
  url = {https://openreview.net/forum?id=JYN2sXXkXxE},
}

Loading related content...

Figure