Abstract
We propose an approach for annotating object classes using free-form text written by undirected and untrained annotators. Free-form labeling is natural for annotators, they intuitively provide very specific and exhaustive labels, and no training stage is necessary. We first collect 729k labels on 15k images using 124 different annotators. Then we automatically enrich the structure of these free-form annotations by discovering a natural vocabulary of 4020 classes within them. This vocabulary represents the natural distribution of objects well and is learned directly from data, instead of being an educated guess done before collecting any labels. Hence, the natural vocabulary emerges from a large mass of free-form annotations. To do so, we (i) map the raw input strings to entities in an ontology of physical objects (which gives them an unambiguous meaning); and (ii) leverage inter-annotator co-occurrences, as well as biases and knowledge specific to individual annotators. Finally, we also automatically extract natural vocabularies of reduced size that have high object coverage while remaining specific. These reduced vocabularies represent the natural distribution of objects much better than commonly used predefined vocabularies. Moreover, they feature more uniform sample distribution over classes.
Introduction
Most object detection datasets today operate under a "closed-world" assumption. Researchers typically define a fixed vocabulary—such as the 80 classes in COCO or the 600 in Open Images—before any data collection begins. This approach introduces several systemic issues. First, the choice of classes is often an arbitrary guess that may not reflect the true diversity of the visual world. Second, it introduces annotator bias; if a specific object (e.g., an "alpaca") is not in the list, annotators are forced to use a generic or incorrect label (e.g., "sheep"). Finally, rare or highly specific objects are frequently ignored, leading to a "long tail" of data that remains uncaptured.
To address these limitations, we propose a "Click & Type" approach where annotators provide free-form text labels. While this method captures richer data, it introduces significant technical challenges, including spelling errors, linguistic ambiguity (e.g., is a "bun" for hair or bread?), and varying levels of expertise among annotators.
The Natural Vocabulary Pipeline
We developed a multi-stage pipeline to transform raw, unstructured text into a structured "Natural Vocabulary." This process is illustrated in Figure 1.
1. Click & Type and Ontology Mapping
Annotators click on an object and type a label without any prior training. These raw strings are then processed through a spell-checker and a Part-of-Speech (PoS) tagger to identify the primary noun. These nouns are subsequently mapped to entities in a massive hierarchy of physical objects based on JFT/Freebase, which contains over 30,000 potential classes.
2. Spatial Disambiguation
A key innovation of our work is the use of spatial clustering and inter-annotator co-occurrence to resolve ambiguity. If multiple annotators label the same spatial point with different terms, these terms can "triangulate" the correct meaning. For example, if one annotator labels a point "bun" and another labels it "food," the system can infer that the object is a bread bun rather than a hair bun.
As shown in Figure 5, we model these relationships using a graph where nodes represent labels and edges represent co-occurrence weights. High weights between an ambiguous term and a more general term (like "food") allow for precise disambiguation.
3. Knowledge Propagation and Optimization
We further refine the labels by leveraging visual features and annotator expertise. Using a ResNet-based feature extractor, the system can "upgrade" generic labels. If a knowledgeable annotator identifies a specific patch as a "Retriever" and a less experienced annotator labels a similar patch as a "Dog," the system propagates the more specific label. Finally, we use dynamic programming to extract reduced vocabularies that maximize object coverage while maintaining high specificity.
Experimental Results
We evaluated our approach using 15,100 images from the Open Images dataset, collecting 729,469 point annotations from 124 diverse annotators across India and the USA.
- Vocabulary Emergence: A natural vocabulary of 4,020 classes emerged, far exceeding standard benchmarks.
- Mapping Accuracy: Our automated pipeline achieved 89% accuracy in mapping raw strings to ontology entities. Post-processing resolved 99.2% of points to unambiguous entities.
- Superior Coverage: Our reduced vocabularies (e.g., an 80-class version) significantly outperformed the COCO 80-class set in both object coverage and specificity.
- Zero-Shot Performance: Models trained on this diverse, specific vocabulary showed improved performance in zero-shot classification tasks because the training data was more descriptive.
Discussion and Conclusion
Our findings suggest that the relationships between objects are best defined by how people naturally label them, rather than by formal biological taxonomies. This "natural" hierarchy often skips technical intermediate categories, reflecting human intuition.
While our method currently relies on a pre-existing ontology and point annotations, future work will explore extending this to full object boundaries (bounding boxes or segmentation) and investigating how cultural and geographical backgrounds further influence naming conventions. By allowing a vocabulary to emerge from the data, we can create datasets that are more exhaustive, specific, and representative of the real world.
Summary updated