FreeGen: Bridging Visual-Linguistic Discrepancies Towards Diffusion-based Pixel-level Data Synthesis

Wenzhuang Wang, Mingcan Ma, Yong Chen, Changqun Xia, Zhenbao Liang, Jia LiAAAI 2025 · OpenReview

Abstract

Training semantic segmentation models typically requires extensive human-annotated datasets, which are costly and time-consuming to produce. While text-to-image diffusion models offer a potential source of synthetic training data, they often suffer from visual-linguistic discrepancies where internal attention maps do not align precisely with object boundaries. This paper presents FreeGen, a two-stage, "real-data-free" framework that recalibrates diffusion model mechanics to synthesize high-quality image-mask pairs. By integrating Attention Self-Rectification (ASR) and an Entity-level Grounding Decoder (EGD), FreeGen achieves performance levels on par with models trained on human-labeled data across multiple benchmarks.

1. Introduction

The advent of large-scale text-to-image diffusion models has revolutionized content generation. However, utilizing these models for pixel-level tasks like semantic segmentation remains challenging. The primary obstacle is the misalignment between the text prompt's attention and the resulting image's spatial features. FreeGen addresses this by introducing a self-driven mechanism to refine these internal representations, effectively turning a generative model into a high-fidelity data annotator.

Figure 1: Overview of FreeGen. The model takes a text prompt and Gaussian noise to generate an image and a corresponding high-quality mask by rectifying internal attention maps (c) compared to standard approaches (a, b).

2. The Visual-Linguistic Discrepancy Problem

In standard diffusion models, cross-attention maps—which link words to pixels—are frequently noisy or "blurry." For instance, the attention for a "dog" might spill over into the background or overlap with a "cat" in the same scene. If these imprecise maps are used as ground-truth labels for training segmentation networks, the resulting models inherit these inaccuracies. Previous solutions often required real-world data for fine-tuning, defeating the purpose of a purely synthetic pipeline.

3. Methodology

FreeGen employs a two-stage training strategy to bridge the gap between text and pixels.

3.1 Stage 1: Attention Self-Rectification (ASR)

The ASR module improves the spatial precision of attention maps by combining self-attention (pixel-to-pixel relationships) with cross-attention (word-to-pixel relationships). This process ensures that the semantic labels provided by the text prompt are anchored to the sharp spatial boundaries identified by the model's internal feature maps. Furthermore, FreeGen introduces a Temporal Fluctuation Factor (TFF) to filter synthetic data. By monitoring the stability of masks across the denoising timesteps, the system can automatically discard "hallucinated" or inconsistent generations.

3.2 Stage 2: Entity-level Grounding Decoder (EGD)

Following the generation of a high-quality synthetic pool, an Entity-level Grounding Decoder is trained. This decoder utilizes a "hardness-aware" curriculum, learning first from clear, simple image-mask pairs before tackling more complex, multi-object scenes. This stage enables the framework to generalize to open-vocabulary settings, identifying objects described in text that may not have been present in the initial training distribution.

Figure 2: The two-stage training strategy of FreeGen. Stage 1 focuses on Attention Self-Rectification to produce synthetic masks, while Stage 2 trains an Entity-level Grounding Decoder using a hardness-aware strategy.

4. Experimental Results

The effectiveness of FreeGen was evaluated by training standard segmentation architectures (e.g., DeepLabV3, Mask2former) solely on its synthetic output.

  • Benchmark Performance: On the VOC 2012 dataset, FreeGen achieved a significant improvement of 9.3% mIoU over previous state-of-the-art synthetic data methods.
  • Open-Vocabulary Capabilities: In zero-shot scenarios, FreeGen reached a record 64.9% mIoU on unseen classes, demonstrating robust generalization.
  • Cross-Domain Generalization: Models trained on FreeGen data showed superior performance when transferred to complex environments like Cityscapes, suggesting that the diversity of synthetic data helps mitigate domain shift issues common in real-world datasets.

5. Conclusion

FreeGen demonstrates that diffusion models possess the latent knowledge required for precise pixel-level tasks, provided their internal attention mechanisms are properly rectified. By eliminating the need for human-labeled real data, FreeGen paves the way for a more scalable, data-centric approach to computer vision training. Future work will explore extending these self-rectification techniques to other spatial tasks such as object detection and depth estimation.

Summary updated

Abstract

Text-to-image diffusion model has inspired research into text-to-data synthesis without human intervention, where spatial attentions correlated with semantic entities in text prompts are primarily interpreted as pseudo-masks. However, these vannila attentions often deliver visual-linguistic discrepancies, in which the associations between image features and entity-level tokens are unstable and divergent, yielding inferior masks for realistic applications, especially in more practical open-vocabulary settings. To tackle this issue, we propose a novel text-guided self-driven generative paradigm, termed FreeGen, which addresses the discrepancies by recalibrating intrinsic visual-linguistic correlations and serves as a real-data-free method to automatically synthesize open-vocabulary pixel-level data for arbitrary entities. Specifically, we first learn an Attention Self-Rectification mechanism to reproject the inherent attention matrices to achieve robust semantic alignment, thereby obtaining class-discriminative masks. A Temporal Fluctuation Factor is present to assess mask quality based on its variation over uniform sampling timesteps, enabling the selection of reliable masks. These masks are then employed as self-supervised signals to support the learning of an Entity-level Grounding Decoder in a self-training manner, thus producing open-vocabulary segmentation results. Extensive experiments show that the existing segmenters trained on FreeGen narrow the performance gap with real data counterparts and remarkably outperform the state-of-the-art methods.
BibTeX
@inproceedings{wang2025freegen,
  author = {Wenzhuang Wang and Mingcan Ma and Yong Chen and Changqun Xia and Zhenbao Liang and Jia Li},
  title = {{FreeGen: Bridging Visual-Linguistic Discrepancies Towards Diffusion-based Pixel-level Data Synthesis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=84nR3tvVnR},
}
Figure