Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video

Shashanka Venkataramanan, Mamshad Nayeem Rizve, João Carreira, Yuki M. Asano, Yannis Avrithis

TL;DR

Can a single two-hour video be as valuable for AI training as the entire ImageNet dataset? This paper introduces DORA (Discover and tRAck), a self-supervised learning method that learns visual representations by tracking objects through long, continuous video streams. Remarkably, pretraining on just one unlabelled "Walking Tour" video allows a model to outperform ImageNet-pretrained counterparts on dense downstream tasks like semantic segmentation and object detection.

What Problem Does This Paper Address?

Modern computer vision relies on massive datasets like ImageNet (1.3M images) or even larger unlabelled sets of billions of images. However, humans don't learn from shuffled, static snapshots; we learn from a continuous, temporal stream of visual information.

Previous attempts to learn from video often treated frames as mere data augmentations or relied on short, curated clips scraped from the web. These methods frequently failed to close the performance gap with image-based pretraining. The authors ask: Are we making the best use of our data? They argue that the temporal continuity of long, uncurated video is an untapped goldmine for data-economical self-supervised learning (SSL).

Methods

The authors propose DORA, a method that "tracks to learn to recognize." It builds upon the DINO (Self-distillation with no labels) framework but adapts it specifically for the temporal nature of video.

  1. Discovery: Instead of random cropping, DORA uses the self-attention heads of a Vision Transformer (ViT) to implicitly "discover" objects in an initial frame.
  2. Refinement via Sinkhorn-Knopp: To ensure the model doesn't just focus on the same prominent object with every head, they use the Sinkhorn-Knopp algorithm to establish correspondences between prototypes and patches, forcing the attention maps to be spatially distinct and diverse.
  3. Tracking: DORA then uses transformer cross-attention to track these discovered objects across subsequent frames in a video clip.
  4. Distillation Loss: These tracked object regions serve as "multi-object crops." The model is trained to match the features of these tracked regions (seen by a "student" network) to the global features of the frame (seen by a "teacher" network).
The DORA architecture: Discovering and tracking objects end-to-end to provide a signal for self-supervised distillation.

Datasets

The paper introduces the Walking Tours (WTours) dataset:

  • Content: 10 high-resolution (4K), hours-long, egocentric videos of people walking through various cities (Amsterdam, Tokyo, Venice, etc.).
  • Key Feature: These are "single-take" videos with no cuts, providing natural scene transitions and lighting changes.
  • Comparison: Unlike movies or YouTube clips, WTours videos are semantically dense, featuring a high number of unique objects per frame, similar to the density found in ImageNet.
Figure 2: Dataset analysis of a WTours video compared with videos from Epic-Kitchens (Damen et al., 2022), AVA (Gu et al., 2018) and two entire movies, concatenated or cropped to match the duration of the WTours video. (a) Lightness vs. time. (b) Number of objects per frame vs. time. (c) Frequency of classes in entire video. (d) Number of shots. Objects detected by Detic (Zhou et al., 2022b), trained on ImageNet-21k.

Key Insights and Ideas

  • The "No-Cut" Advantage: The absence of cinematic cuts is vital. Tracking-based learning breaks down when the scene jumps. The authors show that pretraining on movies (which have frequent cuts) is significantly less effective than pretraining on continuous walking tours.
  • Data Economy: DORA achieves competitive results using only ~200k frames from a single video, compared to the 1.3 million distinct images in ImageNet. This suggests that temporal consistency provides a much stronger supervisory signal than previously thought.
  • Emergent Localization: By forcing the model to track objects to solve the distillation task, the model naturally learns to produce high-quality attention maps that localize objects, even through occlusions.
Visualizing how Sinkhorn-Knopp refinement (bottom row) produces well-separated object masks compared to standard cross-attention (middle row).

Key Results

  • Semantic Segmentation (ADE20k): DORA pretrained on a single video (Venice) achieved 35.4 mIoU, outperforming DINO pretrained on the full ImageNet-1k dataset (33.9 mIoU).
  • Object Detection (MS COCO): DORA (Venice) reached 39.5 mAP, nearly matching ImageNet-pretrained DINO (39.9 mAP) despite using a fraction of the data variety.
  • Video Tasks: DORA showed significant gains in Video Object Segmentation (VOS) and multi-object tracking, proving that the representations learned are inherently aware of temporal dynamics.

Limitations and Future Work

  • Diminishing Returns with Scale: While one video is powerful, moving from 1 video to 10 videos (WTall) provided only modest gains. This suggests we need better ways to aggregate information across diverse long-form videos.
  • Domain Specificity: The current dataset is focused on urban outdoor environments. It remains to be seen how well these representations transfer to very different domains (e.g., medical imaging or top-down satellite views).
  • Computational Cost: While data-economical, processing high-resolution video clips with cross-attention is computationally more intensive than processing static images.

In summary, DORA proves that if you "read" a single video very carefully, you can learn as much about the visual world as someone skimming a million random photos.

Summary updated

Abstract

Self-supervised learning has unlocked the potential of scaling up pretraining to billions of images, since annotation is unnecessary. But are we making the best use of data? How more economical can we be? In this work, we attempt to answer this question by making two contributions. First, we investigate first-person videos and introduce a "Walking Tours" dataset. These videos are high-resolution, hours-long, captured in a single uninterrupted take, depicting a large number of objects and actions with natural scene transitions. They are unlabeled and uncurated, thus realistic for self-supervision and comparable with human learning. Second, we introduce a novel self-supervised image pretraining method tailored for learning from continuous videos. Existing methods typically adapt image-based pretraining approaches to incorporate more frames. Instead, we advocate a "tracking to learn to recognize" approach. Our method called DoRA, leads to attention maps that Discover and tRAck objects over time in an end-to-end manner, using transformer cross-attention. We derive multiple views from the tracks and use them in a classical self-supervised distillation loss. Using our novel approach, a single Walking Tours video remarkably becomes a strong competitor to ImageNet for several image and video downstream tasks.
BibTeX
@inproceedings{venkataramanan2024is,
  author = {Shashanka Venkataramanan and Mamshad Nayeem Rizve and João Carreira and Yuki M. Asano and Yannis Avrithis},
  title = {{Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=Yen1lGns2o},
}
Figure