BAM! The Behance Artistic Media Dataset for Recognition Beyond Photography

Michael J. Wilber, Chen Fang, Hailin Jin, Aaron Hertzmann, John Collomosse, Serge J. Belongie

Introduction

Modern computer vision systems, typically trained on massive photographic datasets like ImageNet or COCO, have achieved remarkable accuracy in recognizing objects in everyday photos. However, these models often falter when encountering the same objects depicted through artistic lenses—such as a pencil sketch of a cat or a watercolor painting of a car. This "representation gap" occurs because art is not bound by physical reality; artists use abstraction, diverse media, and unique styles to reinterpret the world.

To bridge this gap, this research introduces the Behance Artistic Media (BAM!) dataset. BAM! is a large-scale collection of contemporary digital and traditional art annotated for content, artistic media, and emotion, providing a benchmark for teaching machines to understand high-level aesthetics and recognize objects across diverse domains.

The BAM! Dataset

Sourced from 65 million images on Behance, a portfolio platform for professional artists, the BAM! dataset addresses the noise inherent in user-generated tags. While a "cat" tag on Behance might only yield an actual cat 36% of the time, the authors utilized a human-in-the-loop pipeline to ensure high quality.

The final dataset features 20 binary attributes across three facets:

  • Content: Bicycle, bird, building, cars, cat, dog, flower, people, tree.
  • Media: 3D graphics, comic, graphite, oil paint, pen ink, vector art, watercolor.
  • Emotion: Happy, gloomy, scary, peaceful.

Methodology: Scaling Human Effort with AI

To build a dataset of this magnitude without sacrificing precision, the authors employed an active learning pipeline:

  1. Seed & Train: A small set of human-labeled images was used to train initial ResNet (for content) and StyleNet (for media/emotion) models.
  2. Score & Select: The classifier processed the unlabeled dataset, identifying "informative" images (those with high uncertainty) for manual binary labeling by Amazon Mechanical Turk workers.
  3. Iterate & Threshold: After four iterations, strict confidence thresholds were applied to automatically label the remaining dataset, achieving a 90.4% mean precision.

Additionally, the authors introduced the Joint Attribute Model (JAM). This model uses a custom soft-margin loss function to handle non-mutually exclusive labels, allowing it to predict multiple artistic attributes simultaneously.

Key Insights and Results

The Reality of the Representation Gap

Off-the-shelf object detectors like YOLO and SSD suffer massive performance drops when applied to artwork. While the semantic meaning of an object remains constant, the drastic shift in visual representation across media confuses standard models. Interestingly, the research found that iconic objects (like 3D rendered cars) transfer across domains more easily than highly stylized ones (like watercolor cars).

Feature Specialization

Experiments revealed that features from ImageNet-trained models (ResNet) are superior for content recognition, whereas features from style-trained models (StyleNet) are significantly better at predicting emotion and media. This suggests that "content" and "style" require different types of visual feature extraction.

Performance Gains

By fusing standard ImageNet predictions with BAM!-trained classifiers, the researchers significantly improved object detection in art. For example, "Cat" detection Average Precision (AP) rose from ~0.86 to 0.95. Furthermore, pre-training on BAM! improved style classification on external datasets like AVA, moving accuracy from 0.560 to 0.615.

Conclusion and Future Work

The BAM! dataset provides a massive, rich playground for research in domain adaptation and aesthetic understanding. While the Joint Attribute Model successfully learns visual subspaces for content and style, these concepts remain somewhat entangled. Future work will focus on feature disentangling—the ability to perfectly separate an object's identity from the artistic style in which it is rendered.

Summary updated

Abstract

Computer vision systems are designed to work well within the context of everyday photography. However, artists often render the world around them in ways that do not resemble photographs. Artwork produced by people is not constrained to mimic the physical world, making it more challenging for machines to recognize.,,This work is a step toward teaching machines how to categorize images in ways that are valuable to humans. First, we collect a large-scale dataset of contemporary artwork from Behance, a website containing millions of portfolios from professional and commercial artists. We annotate Behance imagery with rich attribute labels for content, emotions, and artistic media. Furthermore, we carry out baseline experiments to show the value of this dataset for artistic style prediction, for improving the generality of existing object classifiers, and for the study of visual domain adaptation. We believe our Behance Artistic Media dataset will be a good starting point for researchers wishing to study artistic imagery and relevant problems. This dataset can be found at https://bam-dataset.org/.
BibTeX
@misc{wilber2017bam,
  author = {Michael J. Wilber and Chen Fang and Hailin Jin and Aaron Hertzmann and John Collomosse and Serge J. Belongie},
  title = {{BAM! The Behance Artistic Media Dataset for Recognition Beyond Photography}},
  year = {2017},
  eprint = {1704.08614},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1704.08614},
}
Figure