Constrained Graphic Layout Generation via Latent Optimization

Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, Kota Yamaguchi

Abstract

It is common in graphic design humans visually arrange various elements according to their design intent and semantics. For example, a title text almost always appears on top of other elements in a document. In this work, we generate graphic layouts that can flexibly incorporate such design semantics, either specified implicitly or explicitly by a user. We optimize using the latent space of an off-the-shelf layout generation model, allowing our approach to be complementary to and used with existing layout generation models. Our approach builds on a generative layout model based on a Transformer architecture, and formulates the layout generation as a constrained optimization problem where design constraints are used for element alignment, overlap avoidance, or any other user-specified relationship. We show in the experiments that our approach is capable of generating realistic layouts in both constrained and unconstrained generation tasks with a single model.

Introduction

Graphic design is a balancing act between aesthetic "realism" and functional constraints. While deep generative models like Generative Adversarial Networks (GANs) have become adept at producing "plausible" layouts, they are notoriously difficult to steer toward specific user requirements. Existing approaches typically fall into two camps: unconstrained models that generate aesthetically pleasing layouts but ignore specific needs (e.g., "place this button on the left"), and constrained models that incorporate rules during training. However, the latter are inflexible; introducing a new constraint type often requires retraining the model from scratch, which is computationally expensive.

This paper introduces CLG-LO (Constrained Layout Generation via Latent Optimization), a framework for flexible, constraint-aware generation. The core idea is to force a generative model to follow specific rules at inference time without breaking the "realism" learned during training.

Methodology

The proposed approach consists of two major components: an improved base model called LayoutGAN++ and a latent optimization strategy.

LayoutGAN++ Architecture

To provide a strong foundation for optimization, the authors improved the original LayoutGAN architecture. They replaced older relational modules with Transformer blocks to better capture complex spatial relationships between elements. Additionally, they introduced an Auxiliary Decoding task for the discriminator. During training, the discriminator must not only distinguish between "real" and "fake" layouts but also attempt to reconstruct the layout. This forces the model to be "position-aware," significantly improving its sensitivity to alignment and spacing.

The LayoutGAN++ architecture utilizes Transformer blocks and an auxiliary decoder to improve spatial awareness.

CLG-LO: Latent Space Exploration

Instead of modifying the model's weights, CLG-LO keeps the pre-trained model frozen and searches for an optimal latent code $z$ that satisfies user constraints. The optimization objective balances two terms: the discriminator's "realism" score and a penalty for violating user-specified constraints (such as alignment or non-overlap).

Overview of the CLG-LO framework: user constraints guide the exploration of the GAN's latent space to produce optimized layouts.

The framework utilizes the Augmented Lagrangian Method to iteratively refine the latent code. While standard gradient descent (Adam) was tested, the authors found that CMA-ES (Covariance Matrix Adaptation Evolution Strategy) was more effective at navigating the complex, non-convex landscape of the GAN's latent space.

Evaluation and Results

The authors evaluated CLG-LO on three diverse datasets: Rico (mobile UI designs), PubLayNet (document layouts), and Magazine (complex page designs).

Key Findings

  • State-of-the-Art Realism: LayoutGAN++ outperformed previous models like NDN and the original LayoutGAN across FID (realism) and IoU (overlap) metrics in unconstrained tasks.
  • Effective Constraint Satisfaction: In "beautification" tasks (fixing overlaps and alignment), CLG-LO reduced the overlap metric from 22.80 in the base model to a nearly perfect 0.02 using the CMA-ES optimizer.
  • Relational Success: When tested with relational constraints (e.g., "Element A is smaller than Element B"), CLG-LO achieved a constraint violation rate of only 0.84% on the Rico dataset, compared to 12.75% for the previous state-of-the-art.

Discussion

The paper demonstrates that a well-trained GAN's latent space acts as a "design manifold" that already contains high-quality layouts. By using the discriminator as part of the optimization objective, the system avoids "adversarial" solutions—layouts that mathematically satisfy constraints but appear disorganized to humans.

However, challenges remain. As the number of user constraints increases, the optimization problem becomes significantly harder, potentially leading to failures if rules conflict. Additionally, while latent optimization is faster than retraining (approx. 1.45s per layout), it may still be too slow for real-time interactive design tools. Future work could explore incorporating human visual saliency models to ensure layouts are not just geometrically correct but also perceptually effective.

Summary updated

Abstract

It is common in graphic design humans visually arrange various elements according to their design intent and semantics. For example, a title text almost always appears on top of other elements in a document. In this work, we generate graphic layouts that can flexibly incorporate such design semantics, either specified implicitly or explicitly by a user. We optimize using the latent space of an off-the-shelf layout generation model, allowing our approach to be complementary to and used with existing layout generation models. Our approach builds on a generative layout model based on a Transformer architecture, and formulates the layout generation as a constrained optimization problem where design constraints are used for element alignment, overlap avoidance, or any other user-specified relationship. We show in the experiments that our approach is capable of generating realistic layouts in both constrained and unconstrained generation tasks with a single model. The code is available at https://github.com/ktrk115/const_layout .
BibTeX
@inproceedings{kikuchi2021constrained,
  author = {Kotaro Kikuchi and Edgar Simo-Serra and Mayu Otani and Kota Yamaguchi},
  title = {{Constrained Graphic Layout Generation via Latent Optimization}},
  booktitle = {ACM Multimedia},
  year = {2021},
  url = {https://openreview.net/forum?id=yFllYYLPhp},
}
Figure