Adaptively Labeling Vision Datasets Via Instance-Level Retrieval

Rishav Mukherji, Brandon Trabucco, Yutong Bai, Maxwell Jones, Ruslan SalakhutdinovOpenReview 2026 · Submission to ICLR 2026 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Human annotations are the backbone of modern computer vision, but they are increasingly recognized as an inefficient resource. They typically capture only a single, fixed view of the rich visual information present in images. How can we move toward datasets that are labeled adaptively, rather than exhaustively by hand? We propose Instance-Level Retrieval, a method for adaptively building object detection datasets from large collections of unlabeled images. Given just a handful of seed examples, our method automatically finds and labels relevant training data by comparing self-supervised object representations. Starting from a small, subset of Pascal VOC (Visual Object Classes), we demonstrate that it is possible to retrieve a high quality set of images. In experiments that control data scale, models trained on our adaptively-labeled data exceed the performance of training on the original Pascal VOC human annotations with a $0.08$ mAP improvement. We use our retrieval method on out of distribution unlabeled images derived from ImageNet-1K, showing that our method can successfully find high quality exemplars for fixed image classes matching the Pascal VOC training set. Training on this expanded training set leads to an additional $0.105$ mAP improvement over baseline. Finally, we show that our methodology is also useful for filtering and selecting high-quality subsets of human-annotated data, yielding a $0.037$ mAP gain compared to uniformly sampled subsets.
BibTeX
@misc{mukherji2026adaptively,
  author = {Rishav Mukherji and Brandon Trabucco and Yutong Bai and Maxwell Jones and Ruslan Salakhutdinov},
  title = {{Adaptively Labeling Vision Datasets Via Instance-Level Retrieval}},
  year = {2026},
  url = {https://openreview.net/forum?id=R6I8P4DuRf},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2026},
}

Loading related content...

Figure