Towards Flexible Multi-modal Document Models

Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, Kota Yamaguchi

Abstract

Graphic design is an intricate process requiring coordinated decisions across layout, typography, and visual content. Traditional AI approaches often rely on task-specific models, which lack the flexibility required for iterative creative workflows. We introduce FlexDM (Flexible Document Model), a unified framework that treats diverse design tasks as a masked field prediction problem. By leveraging an encoder-transformer-decoder architecture on order-less sets of document elements, FlexDM achieves performance competitive with specialized models across layout generation, attribute prediction, and content filling. Our evaluation on the Rico and Crello datasets demonstrates the model's ability to capture complex multi-modal relationships, providing a versatile tool for automated and assisted design.

Introduction

Professional graphic design involves a series of interdependent choices: determining spatial arrangements, selecting harmonious color palettes, and matching typography to visual context. While specialized AI models have succeeded in automating individual components—such as layout generation or font recommendation—they often fail to capture the holistic nature of design. A designer needs a tool capable of fluidly switching between tasks, such as suggesting a background image for existing text or adjusting a layout to accommodate a new element. FlexDM addresses this need by formulating design as a unified problem of understanding relationships between all document components.

Methodology

The core of FlexDM is the formulation of design tasks as masked field prediction. A document is represented as a set of elements, each defined by multiple fields (e.g., type, position, text, image embedding, font, and color).

Architecture

FlexDM utilizes an encoder-transformer-decoder structure to process these multi-modal inputs. As shown in Figure 1, the model projects multi-modal fields into a shared latent space, processes them via transformer blocks to model global dependencies, and decodes them back into original field spaces to predict missing values. Crucially, the model treats elements as an order-less set, allowing it to scale to documents with varying numbers of components.

The FlexDM architecture combines an encoder-transformer-decoder pipeline with a masked field prediction strategy to handle diverse design tasks.

Training and Task Adaptation

The model is trained using a two-stage approach. First, in-domain pre-training on large-scale design datasets using random masking allows the model to learn general structural priors. Second, explicit multi-task learning fine-tunes the model on specific masking patterns that mirror real-world design scenarios, such as "Element Filling" (predicting all attributes of a missing element) or "Attribute Prediction" (predicting style features like font and color).

Experimental Evaluation

FlexDM was evaluated on two primary benchmarks: Rico, containing over 66,000 mobile UI designs, and Crello, featuring 23,000 professional design templates for social media and banners.

Quantitative Performance

FlexDM demonstrates remarkable efficiency, often matching the performance of "Expert" models—versions of the architecture trained exclusively for a single task. On the Crello dataset, FlexDM achieved an attribute prediction score of 0.688, nearly reaching the 0.703 score of a task-specific expert. This indicates that a single unified model can effectively learn the nuances of multiple design domains without significant performance degradation.

Qualitative Results

Qualitative analysis reveals that FlexDM generates contextually aware and aesthetically pleasing designs. As shown in Figure 2, the model successfully predicts harmonious colors, appropriate fonts, and logical placements for text and images based on the surrounding context.

Qualitative results on the Crello dataset demonstrate FlexDM's proficiency in predicting attributes (ATTR), text (TXT), images (IMG), and element positions (POS).

In layout-specific tasks on the Rico dataset, FlexDM outperformed previous state-of-the-art methods, producing element placements that more closely aligned with human-designed ground truths.

Discussion and Conclusion

FlexDM represents a significant step toward holistic document modeling. By unifying disparate design tasks into a single framework, it offers a flexible foundation for creative assistance tools. However, several challenges remain. The current model is not fully generative, as it requires an initial document structure to fill. Additionally, performance can decrease as document complexity grows. Future research may explore diffusion-based formulations to enable unconditional generation and more efficient attention mechanisms to handle highly complex layouts.

Summary updated

Abstract

Creative workflows for generating graphical documents involve complex inter-related tasks, such as aligning elements, choosing appropriate fonts, or employing aesthetically harmonious colors. In this work, we attempt at building a holistic model that can jointly solve many different design tasks. Our model, which we denote by FlexDM, treats vector graphic documents as a set of multi-modal elements, and learns to predict masked fields such as element type, position, styling attributes, image, or text, using a unified architecture. Through the use of explicit multi-task learning and in-domain pre-training, our model can better capture the multi-modal relationships among the different document fields. Experimental results corroborate that our single FlexDM is able to successfully solve a multitude of different design tasks, while achieving performance that is competitive with task-specific and costly baselines.
BibTeX
@inproceedings{inoue2023towards,
  author = {Naoto Inoue and Kotaro Kikuchi and Edgar Simo-Serra and Mayu Otani and Kota Yamaguchi},
  title = {{Towards Flexible Multi-modal Document Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure