PASS: An ImageNet replacement for self-supervised pretraining without humans

Yuki M. Asano, Christian Rupprecht, Andrew Zisserman, Andrea VedaldiNeurIPS 2021 · OpenReview

TL;DR

This paper introduces PASS, a large-scale, ethically-sourced image dataset designed for self-supervised pretraining. PASS replaces ImageNet by excluding all human imagery and relying on permissive licenses, demonstrating that high-quality pretraining is achievable without these problematic elements, even for tasks involving humans.

What Problem Does This Paper Address?

For years, large internet-scraped datasets like ImageNet have been the bedrock of computer vision research, enabling powerful models through pretraining. However, these datasets suffer from significant ethical and technical drawbacks: they often contain personal information without consent, have unclear licensing, exhibit harmful biases, and can include problematic content. As self-supervised learning (SSL) methods advance, reducing the reliance on human-labeled data, the need for cleaner, more ethically sound pretraining datasets becomes critical. This paper addresses the challenge of creating such a dataset that mitigates copyright issues, privacy concerns, and inherent biases, while still being effective for modern SSL pretraining.

Methods

The core contribution is the creation of the PASS dataset. This involved a multi-stage filtering process starting from the massive YFCC-100M dataset:

  1. License Filtering: Images with a CC-BY license were selected.
  2. Automated Filtering: A pipeline using pre-trained models (RetinaFace, Cascade-RCNN) was employed to automatically detect and remove images containing faces, humans, and "not safe for work" (NSFW) content.
  3. Manual Verification: Human annotators further reviewed images to flag any remaining problematic content, including personal information and offensive material.
  4. Duplicate Removal: Standard image duplicate detection tools were used.
  5. User Contribution Balancing: To ensure diversity, the number of images per user was capped, and a subset of 1.4 million images was sampled.

The authors then evaluated PASS by pretraining state-of-the-art SSL models like MoCo-v2, SwAV, and DINO on it. They compared its performance against ImageNet and a human-removed version of ImageNet (IN-1k*) across various downstream tasks, including linear probing, few-shot classification, object detection, and dense pose estimation.

Datasets

  • Training/Pretraining:
    • PASS: The newly introduced dataset, comprising 1.4 million CC-BY licensed images without humans.
  • ImageNet (IN-1k): The standard benchmark dataset for comparison.
  • ImageNet without humans (IN-1k*): An ablation study dataset derived from ImageNet.
  • Places-205: Used for evaluating domain transfer.
  • YFCC-100M: The source dataset for PASS.
  • Downstream Evaluation:
    • ImageNet (IN-1k), ObjectNet, Places205, Flowers, CIFAR-10, Herbarium2019, Pascal VOC07+12, COCO2017, LVIS-v1.

Key Insights and Ideas

  • Ethical Pretraining is Feasible: The most significant insight is that high-performance self-supervised pretraining can be achieved using datasets that are ethically curated and free from problematic content like human imagery.
  • Humans Not Essential for General Pretraining: Removing humans from pretraining datasets has minimal to no negative impact on downstream task performance, even for tasks that do involve humans. This challenges the assumption that diverse human imagery is a prerequisite for robust visual feature learning.
  • PASS as a Viable ImageNet Replacement: PASS demonstrates competitive performance to ImageNet on various transfer learning tasks, suggesting it can serve as a safer and more responsible alternative for pretraining.
  • Bias Mitigation Beyond Humans: While removing humans is a major step, the paper acknowledges that PASS is not entirely bias-free (e.g., geographic bias), highlighting the ongoing challenge of dataset curation.
  • SSL Methods Adapt Well: Modern SSL methods like MoCo-v2, SwAV, and DINO train effectively on PASS, indicating the dataset's suitability for current and future SSL research.

Key Results

  • Competitive Transfer Performance: Models pretrained on PASS achieved performance comparable to, and in some cases superior to, models pretrained on ImageNet (IN-1k and IN-1k*) across various downstream tasks.
  • Object Detection & Segmentation: On COCO, PASS pretraining yielded results within ±1% mAP and AP50 compared to ImageNet pretraining.
  • Human-Centric Tasks: Even on tasks like dense pose estimation, PASS pretraining performed on par with ImageNet pretraining.
  • Frozen Encoder Evaluation: In 8 out of 13 benchmarks, PASS pretraining yielded better results than ImageNet pretraining when transferred to other datasets.
  • Low-Shot Classification: PASS pretraining showed strong performance, often surpassing IN-1k* and competitive with IN-1k on tasks like ObjectNet and Places205.

Limitations and Future Work

  • Geographic Bias: The dataset, like its source YFCC-100M, exhibits a geographic bias, with a focus on Western countries.
  • Harmful Content: Despite rigorous filtering, the authors acknowledge that some harmful content might have slipped through.
  • Not for Human-Specific Tasks: PASS is explicitly designed not to contain humans, making it unsuitable for learning models of people (e.g., pose estimation from scratch).
  • No Labels: PASS cannot be used alone for training and benchmarking tasks that require labels, necessitating curated datasets for such purposes.
  • Future Work: The authors suggest that PASS can serve as a benchmark for studying the effect of image resolution on SSL and for investigating dataset differences in pretraining. They also welcome extensions and contributions to the dataset.

Summary updated

Abstract

Computer vision has long relied on ImageNet and other large datasets of images sampled from the Internet for pretraining models. However, these datasets have ethical and technical shortcomings, such as containing personal information taken without consent, unclear license usage, biases, and, in some cases, even problematic image content. On the other hand, state-of-the-art pretraining is nowadays obtained with unsupervised methods, meaning that labelled datasets such as ImageNet may not be necessary, or perhaps not even optimal, for model pretraining. We thus propose an unlabelled dataset PASS: Pictures without humAns for Self-Supervision. PASS only contains images with CC-BY license and complete attribution metadata, addressing the copyright issue. Most importantly, it contains no images of people at all, and also avoids other types of images that are problematic for data protection or ethics. We show that PASS can be used for pretraining with methods such as MoCo-v2, SwAV and DINO. In the transfer learning setting, it yields similar downstream performances to ImageNet pretraining even on tasks that involve humans, such as human pose estimation. PASS does not make existing datasets obsolete, as for instance it is insufficient for benchmarking. However, it shows that model pretraining is often possible while using safer data, and it also provides the basis for a more robust evaluation of pretraining methods.
BibTeX
@misc{asano2021pass,
  author = {Yuki M. Asano and Christian Rupprecht and Andrew Zisserman and Andrea Vedaldi},
  title = {{PASS: An ImageNet replacement for self-supervised pretraining without humans}},
  year = {2021},
  eprint = {2109.13228},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.13228},
}

Loading related content...

Figure