Abstract: The performance of computer vision models in certain real-world applications (e.g., rare wildlife observation) is limited by the small number of available images. Expanding datasets using pre-trained generative models is an effective way to address this limitation. However, since the automatic generation process is uncontrollable, the generated images are usually limited in diversity, and some of them are undesired. In this paper, we propose a human-guided image generation method for more controllable dataset expansion. We develop a multi-modal projection method with theoretical guarantees to facilitate the exploration of both the original and generated images. Based on the exploration, users refine the prompts and re-generate images for better performance. Since directly refining the prompts is challenging for novice users, we develop a sample-level prompt refinement method to make it easier. With this method, users only need to provide sample-level feedback (e.g., which samples are undesired) to obtain better prompts. The effectiveness of our method is demonstrated through the quantitative evaluation of the multi-modal projection method, improved model performance in the case study for both classification and object detection tasks, and positive feedback from the experts.
1. Introduction
In specialized domains such as rare wildlife observation or medical imaging, data scarcity is a significant hurdle for training robust computer vision models. While generative models like Stable Diffusion can synthesize new training samples, purely automatic expansion often fails. These automated processes typically suffer from two issues: limited diversity, where the model repeats a narrow subset of the training distribution, and undesired images, where semantic ambiguity leads to the generation of irrelevant or anatomically incorrect artifacts.
2. The DataCrafter Framework
We introduce DataCrafter, a visual analytics tool that enables users to audit and refine generated datasets through three core technical pillars:
- Metric Visualization (R1): A dashboard tracks key performance indicators—Informativeness, Diversity, and Distance—allowing users to monitor the quality of the expansion process in real-time.
- Multi-modal Distribution Exploration (R2): To visualize the high-dimensional image space, we employ a Multi-modal Projection (M2M) method. By extracting content labels using LLMs and applying a novel contrastive-learning-based projection, DataCrafter maps images and text labels into a shared 2D scatterplot. This helps users identify "blind spots" in diversity or clusters of low-quality images.
- Sample-level Prompt Refinement (R3): To lower the barrier for domain experts, we replace manual prompt engineering with sample-level feedback. Users simply select undesired images or request more of a specific type. An evolutionary algorithm then optimizes the underlying text prompts to better align with user intent.
3. Key Results and Insights
Evaluation on the Pets and COCO17 datasets demonstrates that human guidance acts as a force multiplier for data expansion:
- Significant Accuracy Gains: On the Pets dataset, classification accuracy rose from a baseline of 48.45% to 81.80% after human-guided expansion, outperforming leading automatic methods by approximately 5%.
- Improved Generalization: The benefits extended to object detection, where mAP increased from 92.4% to 94.5% via pseudo-labeling on the expanded set.
- Mitigating Semantic Drift: A case study on "Bengal" cats revealed that the model initially produced tiger-cat hybrids. Human intervention allowed for prompt refinement (e.g., specifying "Bengal cat" vs "Bengal tiger"), resolving anatomical errors that automated systems could not detect.
4. Conclusion and Future Work
DataCrafter demonstrates that human-in-the-loop systems can overcome the "performance ceiling" often hit by fully automated data expansion methods. While highly effective, the process currently requires human time and is dependent on the quality of the underlying generative model. Future research will explore more complex style-fusing operations and the application of this framework to highly specialized domains like medical imaging.
Summary updated