Abstract
Deep networks thrive when trained on large scale data collections. This has given ImageNet a central role in the development of deep architectures for visual object classification. However, ImageNet was created during a specific period in time, and as such it is prone to aging, as well as dataset bias issues. Moving beyond fixed training datasets will lead to more robust visual systems, especially when deployed on robots in new environments which must train on the objects they encounter there. To make this possible, it is important to break free from the need for manual annotators. Recent work has begun to investigate how to use the massive amount of images available on the Web in place of manual image annotations. We contribute to this research thread with two findings: (1) a study correlating a given level of noisily labels to the expected drop in accuracy, for two deep architectures, on two different types of noise, that clearly identifies GoogLeNet as a suitable architecture for learning from Web data; (2) a recipe for the creation of Web datasets with minimal noise and maximum visual variability, based on a visual and natural language processing concept expansion strategy. By combining these two results, we obtain a method for learning powerful deep object models automatically from the Web. We confirm the effectiveness of our approach through object categorization experiments using our Web-derived version of ImageNet on a popular robot vision benchmark database, and on a lifelong object discovery task on a mobile robot.
Introduction
The success of deep learning in computer vision is inextricably linked to the availability of massive, human-annotated datasets like ImageNet. However, these static benchmarks suffer from "dataset aging"—the gradual divergence between the appearance of objects in the dataset (captured circa 2010) and their appearance in the modern world. For robotics applications, where agents encounter novel objects in dynamic environments, relying on fixed datasets is insufficient.
The World Wide Web offers an inexhaustible supply of visual data, but it is inherently "noisy." Automated searches often return irrelevant images or incorrect labels. This research addresses the challenge of automatically harvesting high-quality, diverse training data from the web to build robust visual models without manual intervention.
Methodology
The Noise Stress Test
To understand how deep architectures handle imperfect data, we conducted a "noise stress test." We intentionally introduced two types of noise into ImageNet: external noise (images from outside the dataset categories) and internal noise (swapping labels between existing classes).
A key discovery was GoogLeNet’s non-linear response to noise: it maintains high accuracy until a "break point" of approximately 35% noise, after which it fails to converge. This characteristic allows GoogLeNet to serve as a diagnostic tool for dataset quality; if the model converges, the noise level is likely manageable.
The WINC Pipeline
We developed the Web ImageNet Cleaned (WINC) pipeline to maximize the utility of web data. The process involves:
- Query Expansion: Using NLP and SEO tools to identify visually related terms (e.g., "purebred" or "arctic" for the category "husky").
- Multi-Engine Harvesting: Collecting the top 200 results from Google, Yahoo, and Bing for each expanded term.
- Cleaning and Selection: Removing duplicates via perceptual hashing and using a scoring function in the feature space of a pre-trained CNN to select the top 5 expansions per class. This ensures a balance between conceptual relevance and visual diversity.
Experimental Results
The effectiveness of the WINC approach was evaluated on the Washington RGB-D robot vision benchmark and a real-world "lifelong discovery" task.
Robot Vision Benchmarking
On the Washington dataset, models trained using the WINC pipeline achieved an accuracy of 87.3%. This is remarkably close to the 88.3% achieved by models trained on the manually curated ImageNet, demonstrating that automated web harvesting can nearly match human-level annotation quality.
Lifelong Object Discovery
In a mobile robot deployment within an office environment, WINC-trained models provided superior semantic category proposals (measured by WUP similarity) compared to standard ImageNet models. The WINC models showed better generalization to the robot's specific environment, likely because the "bias" of the modern web is more aligned with current real-world object appearances than a decade-old dataset.
Conclusion
This work provides a blueprint for bypassing manual annotation in deep learning. By leveraging robust architectures like GoogLeNet and intelligent query expansion strategies, we can harness the scale of the web to build visual models that are both accurate and adaptable to new domains. Future work will explore task-specific database creation and the sensitivity of individual network layers to label noise.
Summary updated