WordArt Designer: User-Driven Artistic Typography Synthesis using Large Language Models

Jun-Yan He, Zhi-Qi Cheng, Chenyang Li, Jingdong Sun, Wangmeng Xiang, Xianhui Lin, Xiaoyang Kang, Zengke Jin, Yusen Hu, Bin Luo, Yifeng Geng, Xuansong Xie, Jingren Zhou

TL;DR

This paper introduces WordArt Designer, a novel framework that leverages Large Language Models (LLMs) to enable users to create artistic typography. It addresses the challenge of making sophisticated typography design accessible to non-professionals by integrating semantic understanding, stylistic transformation, and texture rendering into a user-friendly system.

What Problem Does This Paper Address?

Typography, while crucial in various fields like advertising and design, is often a complex skill requiring professional expertise. Existing tools for typography generation, even those leveraging AI, tend to be limited in their adaptability and creativity, often producing semantically coherent but stylistically constrained results. The paper aims to bridge this gap by empowering users, even those without design backgrounds, to generate unique and aesthetically pleasing artistic typography based on their conceptual inputs. This involves translating abstract ideas into visually rich and semantically relevant typographic designs.

Methods

WordArt Designer employs a four-module architecture, orchestrated by an LLM (specifically GPT-3.5):

  • LLM Engine: This module acts as the core interpreter. It takes user input (e.g., "a cat in jewelry design") and generates actionable prompts for the subsequent modules, effectively translating abstract concepts into concrete design directives.
  • SemTypo Module: This module focuses on semantic typography. It involves:
  • Character Extraction and Parameterization: Natural language input is parsed into a structured format, and character contours are extracted and parameterized using FreeType and a differentiable vector graphic rasterization scheme.
  • Region Selection for Transformation: A novel region-based approach is used to selectively transform parts of characters, aiming to reduce distortions and maintain legibility.
  • Semantic Transformation and Differentiable Rasterization: The transformed parameters are rasterized, and a vision-language backbone model is used with a Score Distillation Sampling (SDS) loss to optimize parameters based on semantic concepts and augmented image batches.
  • StyTypo Module: Building on the semantic layout from SemTypo, this module generates smoother, more detailed stylized images. It uses a depth2image pipeline from Latent Diffusion Models (LDMs) and incorporates a ranking model to select the best stylized results.
  • TexTypo Module: This module enhances the design with texture rendering. It adapts ControlNet by introducing "Scribble conditions" (in addition to Canny Edge and Depth) to encourage more creative and artistic texture generation, moving beyond purely geometric constraints. It also allows for pre-trained diffusion models relevant to specific industries (e.g., advertising, fashion).

The overall workflow is iterative, with the LLM guiding the process and ensuring the final output aligns with user intent and aesthetic appeal.

Datasets

The paper mentions the use of a dataset for training the ranking model within the StyTypo module. This dataset consists of 141 single-character Chinese characters and 5,814 stylized typographic images, classified into "Successful Stylization" and "Failed Stylization." The ResNet18 classification model is used for this task.

Key Insights and Ideas

  • LLMs as Creative Directors: The paper demonstrates the power of LLMs not just for text generation but as sophisticated interpreters and directors for complex visual synthesis tasks, translating abstract user intent into actionable design prompts.
  • Region-Based Semantic Transformation: The novel region-based approach in SemTypo is crucial for maintaining character legibility and reducing distortions, a common issue in typography synthesis, especially for languages with complex characters.
  • Scribble Conditions for Texture: Introducing "Scribble conditions" in ControlNet for texture rendering is a key innovation, allowing for more creative and less geometrically constrained textures, leading to richer artistic outcomes.
  • Modular and Iterative Design: The four-module pipeline, with feedback loops and iterative refinement guided by the LLM, ensures a user-centric approach that prioritizes both creativity and quality.
  • Democratizing Typography: The system's design aims to make advanced artistic typography accessible to a broader audience, fostering creativity beyond professional designers.

Key Results

  • Qualitative Results: The paper showcases impressive qualitative results across various concepts (food, jewelry, animals, etc.) and languages (Chinese and English), demonstrating the system's versatility, creativity, and artistic appeal. Figure 7 and Figure 11 highlight the diverse and "smart" texture rendering.
  • Ranking Model Effectiveness: The ranking model significantly outperforms a random selection method in identifying successfully stylized images, achieving a high success rate (up to 100% for Top10 selected images) and improving overall quality.
  • Deployment: The system has been successfully integrated into ModelScope, utilizing the TongYi Qianwen 14b model as the LLM engine. The deployment details highlight the computational resources and generation times for StyTypo and TexTypo.

Limitations and Future Work

  • Limitations:
    • The paper notes that for abstract nouns and verbs, users may struggle to provide desired descriptions, and the system's LLM is employed to render these into representative objects.
  • While theoretically compatible with all languages, the primary focus and examples are on Chinese and English.
  • Potential ethical concerns are acknowledged, including perpetuating cultural stereotypes, introducing bias, and the risk of including copyrighted graphics.
  • Future Work:
    • The authors aim to further explore the integration of this technology into other design aspects, such as graphics and interactive media.
  • Future work could also involve expanding the system's capabilities to more languages and refining its ability to handle highly abstract concepts.

Summary updated

Abstract

This paper introduces "WordArt Designer", a user-driven framework for artistic typography synthesis, relying on Large Language Models (LLM). The system incorporates four key modules: the "LLM Engine", "SemTypo", "StyTypo", and "TexTypo" modules. 1) The "LLM Engine", empowered by LLM (e.g., GPT-3.5-turbo), interprets user inputs and generates actionable prompts for the other modules, thereby transforming abstract concepts into tangible designs. 2) The "SemTypo module" optimizes font designs using semantic concepts, striking a balance between artistic transformation and readability. 3) Building on the semantic layout provided by the "SemTypo module", the "StyTypo module" creates smooth, refined images. 4) The "TexTypo module" further enhances the design's aesthetics through texture rendering, enabling the generation of inventive textured fonts. Notably, "WordArt Designer" highlights the fusion of generative AI with artistic typography. Experience its capabilities on ModelScope: https://www.modelscope.cn/studios/WordArt/WordArt.
BibTeX
@misc{he2023wordart,
  author = {Jun-Yan He and Zhi-Qi Cheng and Chenyang Li and Jingdong Sun and Wangmeng Xiang and Xianhui Lin and Xiaoyang Kang and Zengke Jin and Yusen Hu and Bin Luo and Yifeng Geng and Xuansong Xie and Jingren Zhou},
  title = {{WordArt Designer: User-Driven Artistic Typography Synthesis using Large Language Models}},
  year = {2023},
  eprint = {2310.18332},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.18332},
}
Figure