TL;DR
Expanding small training datasets using generative models often results in "boring" low-diversity images or bizarre artifacts. DataCrafter is a new human-in-the-loop framework that allows users to visually audit generated data and refine text prompts through simple image-level feedback, significantly boosting the performance of downstream computer vision models.
What Problem Does This Paper Address?
In fields like rare wildlife observation or medical imaging, data is scarce. While we can use generative models (like Stable Diffusion) to "expand" these small datasets, the process is often a "black box."
Automatic expansion suffers from two main issues:
- Limited Diversity: The model might only generate red strawberries because the original sample only had red ones, ignoring the existence of green strawberries.
- Undesired Images: Due to prompt ambiguity, the model might generate a "strawberry-shaped woman" instead of the fruit.
Current tools require users to manually "engineer" complex text prompts to fix these issues—a tedious trial-and-error process that most domain experts (like biologists) aren't trained to do.
Methods
The authors introduce DataCrafter, a visual analytics tool built on three core technical pillars:
- Multi-modal Projection (M2M): To help users see what has been generated, the tool uses GPT-4 to extract "content labels" (e.g., "fur," "grass," "eye") from images. It then uses a novel contrastive-learning-based projection to map both images and these text labels into a 2D scatterplot. This allows users to quickly spot clusters of "bad" images or see where diversity is lacking.
- Sample-level Prompt Refinement: Instead of forcing users to write better code-like prompts, they simply provide feedback on the images (e.g., "delete these weird ones" or "add more like this").
- Evolutionary Prompt Optimization: Behind the scenes, an evolutionary algorithm takes the user's feedback, uses an LLM to mutate the prompt, and generates "proxy images" to see if the new prompt better satisfies the user's goals (like increasing diversity or removing artifacts).
- Metric Visualization: A dashboard tracks three key metrics over time: Informativeness (how useful the image is for training), Diversity, and Distance (how far the generated images stray from the original distribution).
Datasets
The researchers evaluated their method on two popular computer vision benchmarks:
- Pets Dataset: 3,842 images across 37 classes of cats and dogs.
- COCO17: Specifically focusing on 10 animal classes to test expansion capabilities.
- Case Study: They performed a deep dive into the "Bengal" cat class to demonstrate how the tool handles semantic ambiguity (distinguishing between Bengal cats and Bengal tigers).
Key Insights and Ideas
- The "Many-to-Many" Challenge: Most visualization tools assume one label per image. In reality, an image contains many objects (cat, grass, collar). The authors mathematically proved that standard projection methods fail in this "many-to-many" scenario and developed a specific loss function to keep the 2D map organized.
- Semantic Drift: A fascinating "aha" moment occurred during the case study. The generative model initially produced "cat-tiger hybrids." By using DataCrafter, the user realized the prompt "Bengal" was too ambiguous (referring to the region/tiger). Changing the prompt to "Bengal cat" immediately fixed the anatomical errors.
- Human Intuition > Raw Automation: While automatic methods are faster, they often hit a "performance ceiling" where adding more images actually hurts the model. Human guidance allows the dataset to keep growing in quality, not just quantity.
Key Results
The results show that human guidance is a force multiplier for data expansion:
- Accuracy Boost: On the Pets dataset, a model trained on the original small set had 48.45% accuracy. After human-guided expansion with DataCrafter, accuracy jumped to 81.80%.
- Outperforming SOTA: DataCrafter outperformed the leading automatic expansion method (GIF) by approximately 5% in final classification accuracy.
- Generalization: The expanded dataset didn't just help with classification; it also improved Object Detection performance (mAP) from 92.4% to 94.5% using pseudo-labeling.
Limitations and Future Work
- Efficiency vs. Quality: While the tool is more efficient than manual prompt engineering, it still requires a human "in the loop," which is slower than a fully automated pipeline.
- Complex Operations: Currently, the tool handles adding and deleting styles/objects well, but "fusing" complex styles from multiple sets of images is still a work in progress.
- Domain Specificity: The current generative models (Stable Diffusion) are trained on general internet images. The authors note that for highly specialized fields like medical imaging (X-rays), the underlying generative model itself would need better fine-tuning before DataCrafter could be fully effective.
Summary updated