Abstract
The application of computer vision to nuanced subjective use cases is growing. While crowdsourcing has served the vision community well for most objective tasks (such as labeling a "zebra"), it now falters on tasks where there is substantial subjectivity in the concept (such as identifying "gourmet tuna"). However, empowering any user to develop a classifier for their concept is technically difficult: users are neither machine learning experts, nor have the patience to label thousands of examples. In reaction, we introduce the problem of Agile Modeling: the process of turning any subjective visual concept into a computer vision model through a real-time user-in-the-loop interactions. We instantiate an Agile Modeling prototype for image classification and show through a user study (N=14) that users can create classifiers with minimal effort under 30 minutes. We compare this user driven process with the traditional crowdsourcing paradigm and find that the crowd's notion often differs from that of the user's, especially as the concepts become more subjective. Finally, we scale our experiments with simulations of users training classifiers for ImageNet21k categories to further demonstrate the efficacy.
Introduction
In computer vision, identifying objective objects like zebras or stop signs is largely considered a solved problem due to clear majoritarian consensus. However, many real-world applications involve "subjective" concepts—such as "healthy dish" or "gourmet food"—where individual definitions vary significantly. Traditional approaches face two major hurdles: the expertise gap required to build custom classifiers and the immense data requirements of standard models. Agile Modeling addresses these by moving from a "researcher-defined, crowd-labeled" paradigm to a "user-defined, user-labeled" approach that operates in real-time.
Methodology: The Agile Modeling Loop
The Agile Modeling prototype enables non-experts to transform a mental concept into a functional binary classifier through a five-step iterative loop:
- Concept Definition: Users provide initial positive and negative text phrases (e.g., "tuna sushi" vs. "canned tuna").
- Text-to-Image Expansion: Vision-language models (CLIP or ALIGN) identify the top 100 images in a massive unlabeled dataset (e.g., LAION-400M) matching these phrases.
- Interactive Rating: The user labels these 100 images to establish a baseline.
- Model Training: A small Multi-Layer Perceptron (MLP) is trained on pre-computed frozen image embeddings, allowing for near-instantaneous updates.
- Active Learning (AL): The system performs inference on millions of images, selecting those with the highest model uncertainty for the user to label next.
This framework allows for active learning over 10 million images on a single desktop CPU in approximately one minute, making real-time interaction feasible.
Experimental Results
The efficacy of Agile Modeling was evaluated using the LAION-400M dataset for user studies and ImageNet21k for large-scale simulations.
Performance vs. Zero-Shot Baselines
While foundation models like CLIP provide a strong starting point, they often struggle with specific visual nuances. Agile Modeling showed that just 5 minutes of user interaction yielded a 3% boost in AUC PR over zero-shot models. With 24 minutes of interaction (5 rounds of AL), this gain increased to 16%.
The Subjectivity Advantage: User vs. Crowd
A key finding is that the "crowd" is often a poor proxy for individual subjectivity. For nuanced concepts, a model trained on labels from the actual user significantly outperformed one trained on crowd-sourced labels. Specifically, for "hard" subjective concepts, a model with 100 user labels outperformed one with 500 crowd labels, highlighting that data relevance and rater expertise are more critical than raw data volume.
Conclusion and Future Work
Agile Modeling demonstrates that non-experts can build high-quality, custom classifiers in under 30 minutes. However, challenges remain, including user fatigue during labeling and the inherent constraints of the underlying foundation model's embeddings. Future research will focus on reducing the labeling burden through more efficient sampling and expanding the system's ability to handle concepts that are difficult to describe textually.
Summary updated