Unsupervised Graphic Layout Grouping with Transformers

Jialiang Zhu, Danqing Huang, Chunyu Wang, Mingxi Cheng, Ji Li, Han Hu, Xin Geng, Baining Guo

This research, presented at WACV 2023, introduces a novel approach to a fundamental problem in graphic design automation: programmatically understanding the hierarchical structure of a layout.

The Challenge of Layout Grouping

Graphic design is more than just a collection of text and images; it is a structured hierarchy intended to communicate a specific message. Layout grouping is the task of organizing individual elements into perceptual groups, such as a header and its sub-text or an image and its caption.

A complex graphic layout containing text blocks and flow elements that require structural organization.

Traditional methods often rely on "Gestalt principles"—simple heuristics like proximity. However, these rules frequently fail on complex, "overloaded" layouts because they lack global context. Furthermore, training deep learning models for this task is hindered by the scarcity of hierarchical layout datasets, which are incredibly time-consuming for humans to annotate manually.

Methodology: Transformers and Bootstrapping

The authors propose a two-pronged solution involving a new architecture and a clever unsupervised training strategy:

  1. Grouping with Transformers: The model treats layout grouping as a set prediction problem. It uses a Transformer architecture where "group tokens" at different layers learn to represent different levels of the hierarchy, from fine-grained to coarse. An auxiliary "Anchor Prediction" head predicts the bounding box of each group to ground abstract tokens in physical space.
  2. Unsupervised Bootstrapped Training: To bypass the need for human labels, the authors use a two-stage process. They first pre-train on a massive dataset of ~370,000 presentation slides using noisy "pseudo-labels" generated by a simple heuristic. Then, through bootstrapping, the model begins to "self-refine": if the model is highly confident in its own prediction, that prediction replaces the noisy heuristic label for future training rounds.

Key Insights and Results

Unlike simple rules that only look at the distance between neighbors, the Transformer's self-attention mechanism allows every element to "see" the entire slide. This global view is crucial for identifying repetitive patterns or complex alignments that span a whole page.

A comparison demonstrating the original layout (top) versus the model's ability to identify large, coarse structural groups (bottom).

The model significantly outperformed both the heuristic rules it was initialized with and existing state-of-the-art baselines:

  • Quantitative Accuracy: The full model achieved 62.5% soft accuracy (matching at least one level of the hierarchy) and 22.5% strict accuracy (matching the entire hierarchy exactly).
  • Bootstrapping Impact: Bootstrapping alone improved soft accuracy from 40% to 62.5%, proving that the model successfully filtered out the noise in the initial pseudo-labels.
  • Human Evaluation: In a user study, the model's groupings were preferred over baselines with a win-rate exceeding 80%.

Future Directions

While the model excels at geometric organization, the authors identify a lack of semantic understanding as a primary limitation. Currently, the model only looks at the "geometry" of the layout (boxes, colors, types). Incorporating multi-modal features—such as NLP for text meaning and computer vision for image semantics—is the next logical step to resolve ambiguities where visual layout alone is insufficient.

Summary updated

Abstract

Graphic design conveys messages through the combination of text, images and other visual elements. Unstructured designs such as overloaded social media graphics may fail to communicate their intended messages effectively. To address this issue, layout grouping offers a solution by organizing design elements into perceptual groups. While most methods rely on heuristic Gestalt principles, they often lack the context modeling ability needed to handle complex layouts. In this work, we reformulate the layout grouping task as a set prediction problem. It uses Transformers to learn a set of group tokens at various hierarchies, enabling it to reason the membership of the elements more effectively. The self-attention mechanism in Transformers boosts its context modeling ability, which enables it to handle complex layouts more accurately. To reduce annotation costs, we also propose an unsupervised learning strategy that pre-trains on noisy pseudo-labels induced by a novel heuristic algorithm. This approach then bootstraps to self-refine the noisy labels, further improving the accuracy of our model. Our extensive experiments demonstrate the effectiveness of our method, which outperforms existing state-of-the-art approaches in terms of accuracy and efficiency.
BibTeX
@inproceedings{zhu2024unsupervised,
  author = {Jialiang Zhu and Danqing Huang and Chunyu Wang and Mingxi Cheng and Ji Li and Han Hu and Xin Geng and Baining Guo},
  title = {{Unsupervised Graphic Layout Grouping with Transformers}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00107},
}
Figure