Agile Modeling: Image Classification with Domain Experts in the Loop

Otilia Stretcu, Edward Vendrow, Kenji Hata, Krishnamurthy Viswanathan, Vittorio Ferrari, Sasan Tavakkol, Wenlei Zhou, Aditya Avinash, Enming Luo, Neil Gordon Alldrin, MohammadHossein Bateni, Gabriel Berger, Andrew Bunner, Chun-Ta Lu, Javier A Rey, Ariel Fuxman

Abstract

The application of computer vision to nuanced subjective use cases is growing. While crowdsourcing has served the vision community well for most objective tasks (such as labeling a "zebra"), it now falters on tasks where there is substantial subjectivity in the concept (such as identifying "gourmet tuna"). However, empowering any user to develop a classifier for their concept is technically difficult: users are neither machine learning experts, nor have the patience to label thousands of examples. In reaction, we introduce the problem of Agile Modeling: the process of turning any subjective visual concept into a computer vision model through a real-time user-in-the-loop interactions. We instantiate an Agile Modeling prototype for image classification and show through a user study (N=14) that users can create classifiers with minimal effort under 30 minutes. We compare this user driven process with the traditional crowdsourcing paradigm and find that the crowd's notion often differs from that of the user's, especially as the concepts become more subjective. Finally, we scale our experiments with simulations of users training classifiers for ImageNet21k categories to further demonstrate the efficacy.

Introduction

In computer vision, identifying objective objects like zebras or stop signs is largely considered a solved problem due to clear majoritarian consensus. However, many real-world applications involve "subjective" concepts—such as "healthy dish" or "gourmet food"—where individual definitions vary significantly. Traditional approaches face two major hurdles: the expertise gap required to build custom classifiers and the immense data requirements of standard models. Agile Modeling addresses these by moving from a "researcher-defined, crowd-labeled" paradigm to a "user-defined, user-labeled" approach that operates in real-time.

Methodology: The Agile Modeling Loop

The Agile Modeling prototype enables non-experts to transform a mental concept into a functional binary classifier through a five-step iterative loop:

  1. Concept Definition: Users provide initial positive and negative text phrases (e.g., "tuna sushi" vs. "canned tuna").
  2. Text-to-Image Expansion: Vision-language models (CLIP or ALIGN) identify the top 100 images in a massive unlabeled dataset (e.g., LAION-400M) matching these phrases.
  3. Interactive Rating: The user labels these 100 images to establish a baseline.
  4. Model Training: A small Multi-Layer Perceptron (MLP) is trained on pre-computed frozen image embeddings, allowing for near-instantaneous updates.
  5. Active Learning (AL): The system performs inference on millions of images, selecting those with the highest model uncertainty for the user to label next.
The Agile Modeling workflow: from user queries to a trained model via an active learning loop.

This framework allows for active learning over 10 million images on a single desktop CPU in approximately one minute, making real-time interaction feasible.

Experimental Results

The efficacy of Agile Modeling was evaluated using the LAION-400M dataset for user studies and ImageNet21k for large-scale simulations.

Performance vs. Zero-Shot Baselines

While foundation models like CLIP provide a strong starting point, they often struggle with specific visual nuances. Agile Modeling showed that just 5 minutes of user interaction yielded a 3% boost in AUC PR over zero-shot models. With 24 minutes of interaction (5 rounds of AL), this gain increased to 16%.

Performance comparison (AUC PR) across various concepts, demonstrating that Agile Modeling consistently outperforms zero-shot baselines across both CLIP and ALIGN backbones.

The Subjectivity Advantage: User vs. Crowd

A key finding is that the "crowd" is often a poor proxy for individual subjectivity. For nuanced concepts, a model trained on labels from the actual user significantly outperformed one trained on crowd-sourced labels. Specifically, for "hard" subjective concepts, a model with 100 user labels outperformed one with 500 crowd labels, highlighting that data relevance and rater expertise are more critical than raw data volume.

Conclusion and Future Work

Agile Modeling demonstrates that non-experts can build high-quality, custom classifiers in under 30 minutes. However, challenges remain, including user fatigue during labeling and the inherent constraints of the underlying foundation model's embeddings. Future research will focus on reducing the labeling burden through more efficient sampling and expanding the system's ability to handle concepts that are difficult to describe textually.

Summary updated

Abstract

The application of computer vision to nuanced subjective use cases is growing. While crowdsourcing has served the vision community well for most objective tasks (such as labeling a "zebra"), it now falters on tasks where there is substantial subjectivity in the concept (such as identifying "gourmet tuna"). However, empowering any user to develop a classifier for their concept is technically difficult: users are neither machine learning experts, nor have the patience to label thousands of examples. In reaction, we introduce the problem of Agile Modeling: the process of turning any subjective visual concept into a computer vision model through a real-time user-in-the-loop interactions. We instantiate an Agile Modeling prototype for image classification and show through a user study (N=14) that users can create classifiers with minimal effort under 30 minutes. We compare this user driven process with the traditional crowdsourcing paradigm and find that the crowd's notion often differs from that of the user's, especially as the concepts become more subjective. Finally, we scale our experiments with simulations of users training classifiers for ImageNet21k categories to further demonstrate the efficacy.
BibTeX
@inproceedings{stretcu2023agile,
  author = {Otilia Stretcu and Edward Vendrow and Kenji Hata and Krishnamurthy Viswanathan and Vittorio Ferrari and Sasan Tavakkol and Wenlei Zhou and Aditya Avinash and Enming Luo and Neil Gordon Alldrin and MohammadHossein Bateni and Gabriel Berger and Andrew Bunner and Chun-Ta Lu and Javier A Rey and Ariel Fuxman},
  title = {{Agile Modeling: Image Classification with Domain Experts in the Loop}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.02040},
}
Figure