Abstract
Training semantic segmentation models typically requires extensive human-annotated datasets, which are costly and time-consuming to produce. While text-to-image diffusion models offer a potential source of synthetic training data, they often suffer from visual-linguistic discrepancies where internal attention maps do not align precisely with object boundaries. This paper presents FreeGen, a two-stage, "real-data-free" framework that recalibrates diffusion model mechanics to synthesize high-quality image-mask pairs. By integrating Attention Self-Rectification (ASR) and an Entity-level Grounding Decoder (EGD), FreeGen achieves performance levels on par with models trained on human-labeled data across multiple benchmarks.
1. Introduction
The advent of large-scale text-to-image diffusion models has revolutionized content generation. However, utilizing these models for pixel-level tasks like semantic segmentation remains challenging. The primary obstacle is the misalignment between the text prompt's attention and the resulting image's spatial features. FreeGen addresses this by introducing a self-driven mechanism to refine these internal representations, effectively turning a generative model into a high-fidelity data annotator.
2. The Visual-Linguistic Discrepancy Problem
In standard diffusion models, cross-attention maps—which link words to pixels—are frequently noisy or "blurry." For instance, the attention for a "dog" might spill over into the background or overlap with a "cat" in the same scene. If these imprecise maps are used as ground-truth labels for training segmentation networks, the resulting models inherit these inaccuracies. Previous solutions often required real-world data for fine-tuning, defeating the purpose of a purely synthetic pipeline.
3. Methodology
FreeGen employs a two-stage training strategy to bridge the gap between text and pixels.
3.1 Stage 1: Attention Self-Rectification (ASR)
The ASR module improves the spatial precision of attention maps by combining self-attention (pixel-to-pixel relationships) with cross-attention (word-to-pixel relationships). This process ensures that the semantic labels provided by the text prompt are anchored to the sharp spatial boundaries identified by the model's internal feature maps. Furthermore, FreeGen introduces a Temporal Fluctuation Factor (TFF) to filter synthetic data. By monitoring the stability of masks across the denoising timesteps, the system can automatically discard "hallucinated" or inconsistent generations.
3.2 Stage 2: Entity-level Grounding Decoder (EGD)
Following the generation of a high-quality synthetic pool, an Entity-level Grounding Decoder is trained. This decoder utilizes a "hardness-aware" curriculum, learning first from clear, simple image-mask pairs before tackling more complex, multi-object scenes. This stage enables the framework to generalize to open-vocabulary settings, identifying objects described in text that may not have been present in the initial training distribution.
4. Experimental Results
The effectiveness of FreeGen was evaluated by training standard segmentation architectures (e.g., DeepLabV3, Mask2former) solely on its synthetic output.
- Benchmark Performance: On the VOC 2012 dataset, FreeGen achieved a significant improvement of 9.3% mIoU over previous state-of-the-art synthetic data methods.
- Open-Vocabulary Capabilities: In zero-shot scenarios, FreeGen reached a record 64.9% mIoU on unseen classes, demonstrating robust generalization.
- Cross-Domain Generalization: Models trained on FreeGen data showed superior performance when transferred to complex environments like Cityscapes, suggesting that the diversity of synthetic data helps mitigate domain shift issues common in real-world datasets.
5. Conclusion
FreeGen demonstrates that diffusion models possess the latent knowledge required for precise pixel-level tasks, provided their internal attention mechanisms are properly rectified. By eliminating the need for human-labeled real data, FreeGen paves the way for a more scalable, data-centric approach to computer vision training. Future work will explore extending these self-rectification techniques to other spatial tasks such as object detection and depth estimation.
Summary updated