ObjBlur: A Curriculum Learning Approach With Progressive Object-Level Blurring for Improved Layout-to-Image Generation

Stanislav Frolov, Brian Bernhard Moser, Sebastian Palacio, Andreas Dengel

Abstract

Layout-to-image (L2I) generation requires models to synthesize complex scenes from bounding boxes and labels, balancing global coherence with local detail. We introduce ObjBlur, a model-agnostic curriculum learning strategy that progressively reduces object-level blur during training. By starting with low-frequency structural information and gradually introducing high-frequency details, ObjBlur stabilizes the training process and mitigates augmentation leaking. Our experiments on COCO-Stuff and Visual Genome demonstrate that ObjBlur significantly improves image quality and recognizability across GAN and Diffusion-based architectures, achieving a 20% FID improvement on LostGAN and a 34.43% SceneFID improvement on LayoutDiffusion.

1. Introduction

Layout-to-image (L2I) generation is a challenging computer vision task where a model must generate a realistic image based on a provided layout of bounding boxes and semantic labels. This task requires the generator to simultaneously manage global scene composition and the fine-grained textures of individual objects. Current state-of-the-art models, including Generative Adversarial Networks (GANs) and Diffusion models, often struggle with training instability and "augmentation leaking," where training noise or transformations persist in the final generated output.

While curriculum learning (CL)—the practice of training models on increasingly difficult tasks—has been successfully applied to image resolution, its application to the semantic and spatial complexities of L2I generation has remained largely unexplored. We propose ObjBlur, a "plug-and-play" curriculum learning strategy that leverages object-level blurring to guide the model from coarse structural learning to fine-grained detail synthesis.

2. Methodology: The ObjBlur Strategy

ObjBlur is implemented within the data loading pipeline, making it compatible with any L2I architecture. The core innovation lies in applying blur selectively and progressively.

2.1 Object-Level Blurring

Unlike traditional methods that might blur an entire image, ObjBlur utilizes the layout masks provided in L2I datasets. It applies Gaussian blur specifically to foreground objects or the background, controlled by a probability parameter $p_{obj}$. This selective blurring forces the model to respect semantic boundaries and improves the separation between different elements in a scene.

2.2 The Curriculum Schedule

The "difficulty" of the generation task is modulated by the blur strength. At the beginning of training, images are heavily blurred, removing high-frequency details and leaving only low-frequency structural information. This encourages the model to first master the spatial arrangement of the layout.

As training progresses, the blur strength is reduced according to a schedule function $s(t)$. The authors evaluated several functions:

  • Linear: Constant reduction in blur.
  • Step: Periodic drops in blur intensity.
  • Exponential: Rapid initial reduction.
  • Sinusoidal: This schedule was found to be the most effective, providing a gentle "warm-up" phase and a smooth transition toward the final, clean images.

Crucially, because the blur strength reaches zero by the end of the training cycle, the model does not "learn" to produce blurry images, effectively solving the problem of augmentation leaking.

3. Experimental Setup

The effectiveness of ObjBlur was evaluated on two benchmark datasets:

  • COCO-Stuff: Includes 80 "thing" classes and 91 "stuff" classes, providing a balanced mix of objects and backgrounds.
  • Visual Genome: A highly complex dataset with dense layouts (averaging 3 to 30 objects per image) and intricate spatial relationships.

The strategy was integrated into several baseline models, including LostGAN, CAL2IM, and LayoutDiffusion, to demonstrate its model-agnostic nature.

4. Results and Analysis

The inclusion of ObjBlur led to significant quantitative and qualitative improvements across all tested architectures.

4.1 Quantitative Performance

  • Fréchet Inception Distance (FID): On COCO-Stuff, ObjBlur improved the FID of LostGAN from 27.34 to 21.92, representing a ~20% improvement in overall image quality.
  • SceneFID: To measure object-level quality, the researchers calculated FID on cropped objects. LayoutDiffusion saw a 34.43% relative improvement in SceneFID on the COCO dataset.
  • Classification Accuracy Score (CAS): This metric measures how recognizable the generated objects are to a pre-trained classifier. On Visual Genome, LayoutDiffusion’s CAS increased from 36.45 to 40.15.

4.2 Training Stability

A key finding was the reduction in variance between training runs. GANs, which are notoriously sensitive to hyperparameters and initialization, showed much more consistent performance when trained with the ObjBlur curriculum.

5. Discussion and Conclusion

ObjBlur demonstrates that "structure before detail" is a powerful heuristic for L2I generation. By progressively introducing complexity, the model learns to prioritize layout coherence before attempting to synthesize realistic textures.

5.1 Limitations and Future Work

While effective, ObjBlur currently applies a uniform schedule to all objects. Future research could explore dynamic schedules that account for the inherent difficulty of specific classes (e.g., "human" vs. "sky"). Additionally, exploring other forms of degradation, such as additive noise or geometric distortions within a similar curriculum framework, could yield further improvements.

In conclusion, ObjBlur offers a simple yet highly effective method for improving the performance and stability of layout-to-image models, setting a new standard for curriculum learning in generative tasks.

Summary updated

Abstract

We present ObjBlur, a novel curriculum learning approach to improve layout-to-image generation models, where the task is to produce realistic images from layouts composed of boxes and labels. Our method is based on progressive object-level blurring, which effectively stabilizes training and enhances the quality of generated images. This curriculum learning strategy systematically applies varying degrees of blurring to individual objects or the background during training, starting from strong blurring to progressively cleaner images. Our findings reveal that this approach yields significant performance improvements, stabilized training, smoother convergence, and reduced variance between multiple runs. Moreover, our technique demonstrates its versatility by being compatible with generative adversarial networks and diffusion models, underlining its applicability across various generative modeling paradigms. With ObjBlur, we reach new state-of-the-art results on the complex COCO and Visual Genome datasets.
BibTeX
@inproceedings{frolov2024objblur,
  author = {Stanislav Frolov and Brian Bernhard Moser and Sebastian Palacio and Andreas Dengel},
  title = {{ObjBlur: A Curriculum Learning Approach With Progressive Object-Level Blurring for Improved Layout-to-Image Generation}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3680742},
}
Figure