LVIS: A Dataset for Large Vocabulary Instance Segmentation.

Agrim Gupta, Piotr Dollár, Ross B. Girshick

Abstract

Progress on object detection is enabled by datasets that focus the research community's attention on open challenges. This process led us from simple images to complex scenes and from bounding boxes to segmentation masks. In this work, we introduce LVIS (pronounced 'el-vis'): a new dataset for Large Vocabulary Instance Segmentation. We plan to collect 2.2 million high-quality instance segmentation masks for over 1000 entry-level object categories in 164k images. Due to the Zipfian distribution of categories in natural images, LVIS naturally has a long tail of categories with few training samples. Given that state-of-the-art deep learning methods for object detection perform poorly in the low-sample regime, we believe that our dataset poses an important and exciting new scientific challenge. LVIS is available at http://www.lvisdataset.org.

TL;DR

LVIS (pronounced ‘el-vis’) is a massive new dataset designed to push computer vision beyond the "few-category" regime. By providing over 2 million high-quality segmentation masks for 1000+ object categories, it introduces a "long-tail" challenge where many objects have very few training examples, forcing researchers to develop models that can learn from limited data.

What Problem Does This Paper Address?

Most benchmark datasets for object detection and instance segmentation (like COCO or PASCAL VOC) focus on a small number of categories—typically 20 to 80. In these settings, researchers usually have thousands of training examples for every single class.

However, the real world doesn't look like that. Visual categories follow a Zipfian distribution: a few things (like "person" or "chair") are everywhere, but a vast "long tail" of categories (like "sunflower," "unicycle," or "bollard") appear rarely. Current state-of-the-art deep learning models perform excellently when data is abundant but fail miserably in this "low-sample regime."

LVIS provides dense, high-quality instance segmentation masks for over 1000 categories, capturing the complexity of natural scenes.

Before LVIS, there wasn't a large-scale, high-quality benchmark to rigorously test how well algorithms handle this inescapable long-tail distribution.

Methods

The authors introduced several novel organizational and procedural techniques to make a 1000-category dataset feasible:

  • Federated Dataset Design: It is practically impossible to ask annotators to look for 1000 different categories in every single image. Instead, LVIS is built as a "union of smaller constituent datasets." For each category, they define a Positive Set (images where every instance of that category is labeled) and a Negative Set (images confirmed to not contain that category). This allows for fair evaluation without requiring every image to be exhaustively labeled for every class.
  • Six-Stage Annotation Pipeline: To ensure high quality and "vocabulary recall," they used a rigorous process involving iterative object spotting, exhaustive marking, segmentation, verification, full recall verification, and negative labeling.
  • WordNet Integration: To handle synonyms (e.g., "backpack" vs. "rucksack") and hierarchical relationships, they mapped categories to WordNet synsets.
The dataset uses WordNet to manage synonyms and hierarchical relationships between categories.

Datasets

  • LVIS (The New Benchmark): Features 164k images with a goal of 2.2 million high-quality instance masks across 1000+ categories.
  • COCO: LVIS uses the same images as the COCO 2017 dataset but provides much higher-quality masks and a vastly expanded vocabulary.
  • Comparisons: The paper compares LVIS against ADE20K (which has many categories but fewer images/lower consistency) and Open Images (which has many images but lacks segmentation masks and suffers from center-bias).

Key Insights and Ideas

  • The "Evaluation-First" Principle: The authors designed the evaluation metric (COCO-style Average Precision) first, then built the annotation pipeline specifically to satisfy the requirements of that metric.
  • Quality as a Bottleneck: Coarse masks eventually limit our ability to distinguish between a good algorithm and a great one. LVIS masks are significantly more detailed, following object boundaries much more closely.
  • The Long Tail is a Feature: Rather than trying to balance the dataset artificially, LVIS embraces the natural imbalance of the world, making it a frontier dataset for few-shot learning.

Key Results

  • Heavier Tail: LVIS has a significantly more challenging distribution than previous datasets. The median number of instances per category in their analysis subset was only 9.
  • Superior Mask Quality: LVIS masks achieved higher IoU (Intersection over Union) and boundary quality scores than both COCO and ADE20K when compared to expert annotators.
  • The "Low-Shot" Gap: A standard Mask R-CNN trained on a subset of 1k images drops from 36.4% Mask AP to a mere 9.8%. This quantitative gap proves that LVIS is a significantly harder and more "open" problem for current AI.

Limitations and Future Work

  • Work in Progress: At the time of publication, the full 2.2M mask count was a projection.
  • Hierarchical Evaluation: While they use WordNet, the current evaluation treats all categories as "flat." Future work could involve metrics that account for hierarchical relationships (e.g., a "Canoe" is also a "Boat").
  • Model Performance: Standard models currently perform poorly on rare classes, leaving the task of solving the long-tail problem to the research community.

Summary updated

Abstract

Progress on object detection is enabled by datasets that focus the research community's attention on open challenges. This process led us from simple images to complex scenes and from bounding boxes to segmentation masks. In this work, we introduce LVIS (pronounced 'el-vis'): a new dataset for Large Vocabulary Instance Segmentation. We plan to collect 2.2 million high-quality instance segmentation masks for over 1000 entry-level object categories in 164k images. Due to the Zipfian distribution of categories in natural images, LVIS naturally has a long tail of categories with few training samples. Given that state-of-the-art deep learning methods for object detection perform poorly in the low-sample regime, we believe that our dataset poses an important and exciting new scientific challenge. LVIS is available at http://www.lvisdataset.org.
BibTeX
@inproceedings{gupta2019lvis,
  author = {Agrim Gupta and Piotr Dollár and Ross B. Girshick},
  title = {{LVIS: A Dataset for Large Vocabulary Instance Segmentation.}},
  booktitle = {CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=rQ0wdm7gd6r},
}
Figure