Abstract
Graphic design is an intricate process requiring coordinated decisions across layout, typography, and visual content. Traditional AI approaches often rely on task-specific models, which lack the flexibility required for iterative creative workflows. We introduce FlexDM (Flexible Document Model), a unified framework that treats diverse design tasks as a masked field prediction problem. By leveraging an encoder-transformer-decoder architecture on order-less sets of document elements, FlexDM achieves performance competitive with specialized models across layout generation, attribute prediction, and content filling. Our evaluation on the Rico and Crello datasets demonstrates the model's ability to capture complex multi-modal relationships, providing a versatile tool for automated and assisted design.
Introduction
Professional graphic design involves a series of interdependent choices: determining spatial arrangements, selecting harmonious color palettes, and matching typography to visual context. While specialized AI models have succeeded in automating individual components—such as layout generation or font recommendation—they often fail to capture the holistic nature of design. A designer needs a tool capable of fluidly switching between tasks, such as suggesting a background image for existing text or adjusting a layout to accommodate a new element. FlexDM addresses this need by formulating design as a unified problem of understanding relationships between all document components.
Methodology
The core of FlexDM is the formulation of design tasks as masked field prediction. A document is represented as a set of elements, each defined by multiple fields (e.g., type, position, text, image embedding, font, and color).
Architecture
FlexDM utilizes an encoder-transformer-decoder structure to process these multi-modal inputs. As shown in Figure 1, the model projects multi-modal fields into a shared latent space, processes them via transformer blocks to model global dependencies, and decodes them back into original field spaces to predict missing values. Crucially, the model treats elements as an order-less set, allowing it to scale to documents with varying numbers of components.
Training and Task Adaptation
The model is trained using a two-stage approach. First, in-domain pre-training on large-scale design datasets using random masking allows the model to learn general structural priors. Second, explicit multi-task learning fine-tunes the model on specific masking patterns that mirror real-world design scenarios, such as "Element Filling" (predicting all attributes of a missing element) or "Attribute Prediction" (predicting style features like font and color).
Experimental Evaluation
FlexDM was evaluated on two primary benchmarks: Rico, containing over 66,000 mobile UI designs, and Crello, featuring 23,000 professional design templates for social media and banners.
Quantitative Performance
FlexDM demonstrates remarkable efficiency, often matching the performance of "Expert" models—versions of the architecture trained exclusively for a single task. On the Crello dataset, FlexDM achieved an attribute prediction score of 0.688, nearly reaching the 0.703 score of a task-specific expert. This indicates that a single unified model can effectively learn the nuances of multiple design domains without significant performance degradation.
Qualitative Results
Qualitative analysis reveals that FlexDM generates contextually aware and aesthetically pleasing designs. As shown in Figure 2, the model successfully predicts harmonious colors, appropriate fonts, and logical placements for text and images based on the surrounding context.
In layout-specific tasks on the Rico dataset, FlexDM outperformed previous state-of-the-art methods, producing element placements that more closely aligned with human-designed ground truths.
Discussion and Conclusion
FlexDM represents a significant step toward holistic document modeling. By unifying disparate design tasks into a single framework, it offers a flexible foundation for creative assistance tools. However, several challenges remain. The current model is not fully generative, as it requires an initial document structure to fill. Additionally, performance can decrease as document complexity grows. Future research may explore diffusion-based formulations to enable unconditional generation and more efficient attention mechanisms to handle highly complex layouts.
Summary updated