Introduction
Modern computer vision relies heavily on labeled data. While human-annotated datasets like ImageNet have propelled the field forward, they are expensive to scale and often miss the rich contextual relationships between objects, scenes, and attributes. Furthermore, purely automated semi-supervised learning (SSL) systems often suffer from "semantic drift," where small initial errors compound over time.
NEIL (Never Ending Image Learner) addresses these challenges by acting as an autonomous system that crawls the web 24/7. It builds a massive, structured visual knowledge base by jointly discovering common-sense relationships and labeling image instances. By exploiting the inherent structure of the visual world, NEIL scales learning far beyond human-annotated limits while resisting semantic drift.
The NEIL Framework
NEIL operates on the principle of "Macro-vision," focusing on statistically significant patterns across massive image collections rather than perfect understanding of individual images. The system follows a continuous, iterative cycle to refine its knowledge.
The cycle consists of several key stages:
- Seeding: Starting with a fixed vocabulary of objects, scenes, and attributes, NEIL downloads initial images from search engines.
- Visual Cluster Discovery: To handle search noise (e.g., "Apple" the fruit vs. the company), NEIL uses clustering techniques like exemplar-LDA and Affinity Propagation to identify representative prototypes.
- Relationship Extraction: The system identifies four types of common-sense connections: Object-Object (e.g., "Wheel is part of Car"), Object-Attribute (e.g., "Sunflower is Yellow"), Scene-Object (e.g., "Car is found in Raceway"), and Scene-Attribute (e.g., "Alley is Narrow").
- Retraining with Constraints: Discovered relationships act as regularizers. If NEIL identifies a "car," it verifies the presence of related components like "wheels" or appropriate contexts like a "raceway," effectively self-correcting its labels.
Scaling Visual Knowledge
NEIL's power lies in treating the physical world's structure as a regularizer. By learning that "buildings have windows" or "tigers are found in zoos," the system creates a self-correcting feedback loop. This joint learning of labels and relationships makes the system more robust than learning them in isolation.
The system was trained on over 2 million images scraped from the web, using an initial ontology of 1,152 object categories, 1,034 scene categories, and 87 attributes. Evaluation on independent Flickr datasets confirmed that NEIL generalizes well beyond its initial search engine sources.
Performance and Findings
After running for 2.5 months on a 200-core cluster, NEIL achieved significant milestones in visual knowledge extraction.
Key results include:
- Knowledge Base Growth: NEIL discovered over 1,700 relationships and labeled 400,000 visual instances, including 300,000 with bounding boxes.
- Accuracy Gains: In scene classification, performance improved from a baseline mAP of 0.52 to 0.62. Object detection mAP rose from 0.34 to 0.51, outperforming standard bootstrapping.
- Human Validation: Testers found that 79% of discovered relationships and 98% of visual labels were correct by the sixth iteration.
Conclusion and Future Directions
NEIL demonstrates the feasibility of autonomous, large-scale visual learning. While currently limited by a fixed vocabulary and static images, future iterations could integrate with language learners like NELL to expand its ontology or incorporate video data to learn temporal relationships and actions. Despite the high computational cost, NEIL represents a major step toward AI that learns about the world contextually and continuously.
Summary updated