NEIL: Extracting Visual Knowledge from Web Data

Xinlei Chen, Abhinav Shrivastava, Abhinav Gupta

Introduction

Modern computer vision relies heavily on labeled data. While human-annotated datasets like ImageNet have propelled the field forward, they are expensive to scale and often miss the rich contextual relationships between objects, scenes, and attributes. Furthermore, purely automated semi-supervised learning (SSL) systems often suffer from "semantic drift," where small initial errors compound over time.

NEIL (Never Ending Image Learner) addresses these challenges by acting as an autonomous system that crawls the web 24/7. It builds a massive, structured visual knowledge base by jointly discovering common-sense relationships and labeling image instances. By exploiting the inherent structure of the visual world, NEIL scales learning far beyond human-annotated limits while resisting semantic drift.

The NEIL Framework

NEIL operates on the principle of "Macro-vision," focusing on statistically significant patterns across massive image collections rather than perfect understanding of individual images. The system follows a continuous, iterative cycle to refine its knowledge.

The iterative learning cycle of NEIL, from seed images to relationship discovery and retraining.

The cycle consists of several key stages:

  1. Seeding: Starting with a fixed vocabulary of objects, scenes, and attributes, NEIL downloads initial images from search engines.
  2. Visual Cluster Discovery: To handle search noise (e.g., "Apple" the fruit vs. the company), NEIL uses clustering techniques like exemplar-LDA and Affinity Propagation to identify representative prototypes.
  3. Relationship Extraction: The system identifies four types of common-sense connections: Object-Object (e.g., "Wheel is part of Car"), Object-Attribute (e.g., "Sunflower is Yellow"), Scene-Object (e.g., "Car is found in Raceway"), and Scene-Attribute (e.g., "Alley is Narrow").
  4. Retraining with Constraints: Discovered relationships act as regularizers. If NEIL identifies a "car," it verifies the presence of related components like "wheels" or appropriate contexts like a "raceway," effectively self-correcting its labels.

Scaling Visual Knowledge

NEIL's power lies in treating the physical world's structure as a regularizer. By learning that "buildings have windows" or "tigers are found in zoos," the system creates a self-correcting feedback loop. This joint learning of labels and relationships makes the system more robust than learning them in isolation.

The system was trained on over 2 million images scraped from the web, using an initial ontology of 1,152 object categories, 1,034 scene categories, and 87 attributes. Evaluation on independent Flickr datasets confirmed that NEIL generalizes well beyond its initial search engine sources.

Performance and Findings

After running for 2.5 months on a 200-core cluster, NEIL achieved significant milestones in visual knowledge extraction.

Examples of visual instances and common-sense relationships extracted by NEIL.

Key results include:

  • Knowledge Base Growth: NEIL discovered over 1,700 relationships and labeled 400,000 visual instances, including 300,000 with bounding boxes.
  • Accuracy Gains: In scene classification, performance improved from a baseline mAP of 0.52 to 0.62. Object detection mAP rose from 0.34 to 0.51, outperforming standard bootstrapping.
  • Human Validation: Testers found that 79% of discovered relationships and 98% of visual labels were correct by the sixth iteration.

Conclusion and Future Directions

NEIL demonstrates the feasibility of autonomous, large-scale visual learning. While currently limited by a fixed vocabulary and static images, future iterations could integrate with language learners like NELL to expand its ontology or incorporate video data to learn temporal relationships and actions. Despite the high computational cost, NEIL represents a major step toward AI that learns about the world contextually and continuously.

Summary updated

Abstract

We propose NEIL (Never Ending Image Learner), a computer program that runs 24 hours per day and 7 days per week to automatically extract visual knowledge from Internet data. NEIL uses a semi-supervised learning algorithm that jointly discovers common sense relationships (e.g., "Corolla is a kind of/looks similar to Car", "Wheel is a part of Car") and labels instances of the given visual categories. It is an attempt to develop the world's largest visual structured knowledge base with minimum human labeling effort. As of 10th October 2013, NEIL has been continuously running for 2.5 months on 200 core cluster (more than 350K CPU hours) and has an ontology of 1152 object categories, 1034 scene categories and 87 attributes. During this period, NEIL has discovered more than 1700 relationships and has labeled more than 400K visual instances.
BibTeX
@inproceedings{chen2013neil,
  author = {Xinlei Chen and Abhinav Shrivastava and Abhinav Gupta},
  title = {{NEIL: Extracting Visual Knowledge from Web Data}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2013},
  doi = {10.1109/iccv.2013.178},
}
Figure