Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video

Shashanka Venkataramanan, Mamshad Nayeem Rizve, João Carreira, Yuki M. Asano, Yannis Avrithis

Abstract

Self-supervised learning has unlocked the potential of scaling up pretraining to billions of images, since annotation is unnecessary. But are we making the best use of data? How more economical can we be? In this work, we attempt to answer this question by making two contributions. First, we investigate first-person videos and introduce a "Walking Tours" dataset. These videos are high-resolution, hours-long, captured in a single uninterrupted take, depicting a large number of objects and actions with natural scene transitions. They are unlabeled and uncurated, thus realistic for self-supervision and comparable with human learning. Second, we introduce a novel self-supervised image pretraining method tailored for learning from continuous videos. Existing methods typically adapt image-based pretraining approaches to incorporate more frames. Instead, we advocate a "tracking to learn to recognize" approach. Our method called DoRA, leads to attention maps that Discover and tRAck objects over time in an end-to-end manner, using transformer cross-attention. We derive multiple views from the tracks and use them in a classical self-supervised distillation loss. Using our novel approach, a single Walking Tours video remarkably becomes a strong competitor to ImageNet for several image and video downstream tasks.

Introduction

Modern computer vision relies heavily on "pretraining"—teaching models basic visual concepts before fine-tuning them for specific tasks. For years, the gold standard has been ImageNet, a dataset of 1.3 million carefully curated and labeled images. However, this approach is fundamentally "un-human." Humans do not learn to see by looking at millions of shuffled snapshots; we observe a continuous, uninterrupted stream of visual data.

A sample frame from a walking tour video showing a complex urban scene with a tram and pedestrians.

Previous attempts to learn from video often struggled to match ImageNet’s performance because they treated video frames as random collections of images or used "object-centric" clips that lack real-world complexity. This research asks: Can we bridge this gap by learning from long, first-person, uncurated video?

Methods: DORA (Discovery and tRAcking)

The authors introduce DORA, a self-supervised learning framework designed specifically for continuous video. DORA builds on the DINO (Self-Distillation with No Labels) architecture but adds a crucial temporal dimension. While standard models might look at two random crops of the same image, DORA uses a "tracking to learn to recognize" philosophy:

  1. Discovery: It uses the attention mechanism of a Vision Transformer (ViT) to identify potential objects in a frame.
  2. Refinement: It applies the Sinkhorn-Knopp algorithm to ensure the model's attention is spread across distinct, non-overlapping objects.
  3. Tracking: It tracks these discovered objects across subsequent frames using cross-attention.
  4. Distillation: A teacher-student setup is used where the student network predicts the features of a masked, tracked object based on the teacher's view of the whole scene.

By tracking objects as they move, rotate, or are partially hidden, the model learns that an object is the "same thing" over time, building a robust internal representation of the world.

The Walking Tours (WT) Dataset

The paper introduces the Walking Tours (WT) dataset, which consists of 10 high-resolution (4K), 60fps, first-person videos of people walking through urban scenes like Venice, Amsterdam, and Bangkok.

Representative frames from the Walking Tours (WT) dataset, featuring diverse urban environments and lighting conditions.

Unlike most datasets, these have zero shot-cuts, no curation, and natural transitions in lighting and scenery. The authors found that "cuts" (common in movies or YouTube clips) actually hurt the learning process. Continuous, egocentric video is superior because it forces the model to deal with the persistence of objects over time.

Key Results

The researchers demonstrated that quality and continuity can trump quantity:

  • Beating ImageNet: DORA pretrained on just the 2-hour Venice video achieved 35.4% mIoU on ADE20k segmentation, outperforming a DINO model trained on the entire ImageNet dataset (33.9% mIoU).
  • Detection Parity: On MS COCO object detection, the Venice-trained model reached 39.5 mAP, nearly identical to the ImageNet-trained model's 39.9 mAP.
  • Efficiency: When using the full 10-video Walking Tours dataset, DORA outperformed ImageNet-pretrained models across almost every metric, despite using only about one-tenth the total number of unique frames.
  • Robustness: The model showed a remarkable ability to track objects through occlusions without explicit supervision.

Limitations and Future Work

While the data is "cheaper" to acquire, processing high-resolution video for thousands of iterations requires significant GPU power. Additionally, the current dataset is primarily urban; future work should explore specialized domains like medical imaging or deep-sea exploration. The authors also suggest scaling this to thousands of hours of video and integrating action understanding (verbs) alongside object recognition (nouns).

Summary updated

Abstract

Self-supervised learning has unlocked the potential of scaling up pretraining to billions of images, since annotation is unnecessary. But are we making the best use of data? How more economical can we be? In this work, we attempt to answer this question by making two contributions. First, we investigate first-person videos and introduce a "Walking Tours" dataset. These videos are high-resolution, hours-long, captured in a single uninterrupted take, depicting a large number of objects and actions with natural scene transitions. They are unlabeled and uncurated, thus realistic for self-supervision and comparable with human learning. Second, we introduce a novel self-supervised image pretraining method tailored for learning from continuous videos. Existing methods typically adapt image-based pretraining approaches to incorporate more frames. Instead, we advocate a "tracking to learn to recognize" approach. Our method called DoRA, leads to attention maps that Discover and tRAck objects over time in an end-to-end manner, using transformer cross-attention. We derive multiple views from the tracks and use them in a classical self-supervised distillation loss. Using our novel approach, a single Walking Tours video remarkably becomes a strong competitor to ImageNet for several image and video downstream tasks.
BibTeX
@inproceedings{venkataramanan2024is,
  author = {Shashanka Venkataramanan and Mamshad Nayeem Rizve and João Carreira and Yuki M. Asano and Yannis Avrithis},
  title = {{Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=Yen1lGns2o},
}
Figure