Abstract
Progress on object detection is enabled by datasets that focus the research community's attention on open challenges. This process led us from simple images to complex scenes and from bounding boxes to segmentation masks. In this work, we introduce LVIS (pronounced 'el-vis'): a new dataset for Large Vocabulary Instance Segmentation. We plan to collect ~2 million high-quality instance segmentation masks for over 1000 entry-level object categories in 164k images. Due to the Zipfian distribution of categories in natural images, LVIS naturally has a long tail of categories with few training samples. Given that state-of-the-art deep learning methods for object detection perform poorly in the low-sample regime, we believe that our dataset poses an important and exciting new scientific challenge.
Introduction: The Long Tail Challenge
Most computer vision benchmarks, such as COCO or PASCAL VOC, concentrate on a limited set of common object categories where training data is plentiful. However, real-world visual data follows a Zipfian distribution: while a few categories are ubiquitous, a vast "long tail" of objects appears infrequently. Current state-of-the-art models excel with thousands of examples but falter in the low-sample regime. LVIS is designed to push the field beyond the 80 common categories of COCO, challenging researchers to develop algorithms capable of recognizing thousands of categories, even when data is scarce.
The LVIS Dataset
LVIS is built upon 164,000 images from the COCO 2017 set, aiming for approximately 2 million high-quality masks across 1000+ categories. Unlike other large-scale datasets, LVIS masks are noted for their high quality and boundary complexity, following object contours much more closely than previous benchmarks.
A key insight from the dataset's creation is that the "long tail" is inescapable; simply increasing the number of images does not solve data scarcity but instead reveals even rarer categories. This necessitates algorithmic innovation rather than just data collection. Furthermore, LVIS was annotated entirely by humans without "machine-in-the-loop" assistance, ensuring the benchmark does not inherit the biases of current detectors.
Annotation Methodology
The logistical challenge of labeling 1000+ categories was addressed through a Federated Dataset Design. Instead of exhaustive labeling of every image for every category, LVIS is treated as a union of smaller datasets. For each category, a "positive set" (exhaustively labeled) and a "negative set" (confirmed absence) are identified, making the workload manageable.
The annotation follows a rigorous six-stage pipeline: iterative object spotting, exhaustive marking of instances, precise segmentation, quality verification, full recall verification, and negative labeling. To manage the complex vocabulary, the authors utilized WordNet synsets to handle synonyms and hierarchical relationships.
Key Results and Benchmarks
Initial evaluations using standard models like Mask R-CNN reveal a significant performance gap. While achieving an overall Average Precision (AP) of ~21%, performance on "rare" categories was initially as low as 0.8% AP. The authors demonstrated that "Repeat Factor Sampling"—oversampling images with rare objects—could significantly boost rare category performance to 13.4% AP.
Statistical analysis also revealed that AP can be upwardly biased on very small sets of positive examples, a crucial finding for evaluating performance in the long tail.
Conclusion and Future Work
LVIS serves as a scientific challenge to the community, highlighting the limitations of current models in the face of imbalanced data. While techniques like resampling provide a head start, a substantial gap remains between frequent and rare object recognition. Future work will focus on completing the full v1.0 release and encouraging research into few-shot learning, meta-learning, and more robust loss functions for large-vocabulary instance segmentation.
Summary updated