Generating Animated Layouts as Structured Text Representations

Yeonsang Shin, Jihwan Kim, Yumin Song, Kyungseung Lee, HyunHee Chung, Taeyoung Na

TL;DR

This paper introduces "Animated Layout Generation," a new method for creating dynamic video advertisements from text prompts. It tackles the challenge of precisely controlling text and animated graphics by proposing a "Structured Text Representation" (ST-Representation) and a pipeline called VAKER, which leverages LLMs for fine-grained control over temporal dynamics and visual elements. VAKER significantly outperforms existing methods in generating realistic and controllable video advertisements.

What Problem Does This Paper Address?

Generating high-quality video advertisements with precise control over text and animated graphics is a significant challenge. While text-to-video models have advanced, they often struggle with rendering readable text and accurately placing and animating graphical elements. This is particularly problematic for applications like social media videos, promotional content, and informational videos where dynamic visuals and synchronized text are crucial. Existing methods lack the fine-grained control needed to manage spatial composition and temporal evolution of multiple visual elements, especially when dealing with embedded text and animated overlays.

Methods

The core of the proposed solution is Animated Layout Generation, which extends traditional static graphic layouts by incorporating temporal dynamics. This is achieved through a novel Structured Text Representation (ST-Representation). This representation encodes video layouts as structured text, allowing for:

  • Hierarchical Organization: Visual elements are organized hierarchically, preserving spatial and temporal relationships.
  • LLM Compatibility: The text-based format naturally integrates with Large Language Models (LLMs), enabling sophisticated reasoning and generation.
  • Precise Control: It allows for fine-grained control over on-screen text, spatial positioning, and animated elements.

The paper introduces VAKER (Video Ad maKER), a text-to-video advertisement generation pipeline built upon this ST-Representation. VAKER employs a three-stage generation process:

  1. Banner Generation: Creates the banner elements (top, bottom, left, right).
  2. Mainground Generation: Generates the main scene's background, logos, and text.
  3. Animation Generation: Defines the temporal dynamics and movements of foreground objects.

Each stage is handled by a specialized expert model fine-tuned with LoRA. Additionally, Unstructured Text (UT) Reasoning is incorporated, inspired by Chain-of-Thought prompting, to translate natural language prompts into detailed descriptions before generating structured text, bridging the gap between user input and the ST-Representation.

Datasets

The paper utilizes a newly constructed dataset of 2,224 real video advertisements for training and evaluation. This dataset was created using an automated pipeline that extracts key visual information from videos and converts them into three types of data: Structured Text (ST), Unstructured Text (UT) Reasoning, and natural language prompts. The dataset encompasses diverse commercial categories and provides a wide spectrum of advertising layouts and visual design patterns.

Key Insights and Ideas

  • Structured Text as the Key: The paper highlights that representing video layouts as structured text is crucial for enabling fine-grained control, especially for dynamic elements and embedded text. This approach unlocks the power of LLMs for video generation.
  • Decomposition for Control: Breaking down video advertisement generation into distinct stages (Banner, Mainground, Animation) allows for specialized models and better control over each aspect of the final output.
  • UT Reasoning for Robustness: The UT Reasoning component, inspired by CoT prompting, proves vital for translating ambiguous natural language prompts into precise structured specifications, leading to more accurate and stable generation.
  • Animation from Structure: The ability to generate diverse animation patterns, some emerging without explicit training examples, demonstrates the power of the ST-Representation in capturing and translating motion information.

Key Results

  • Quantitative Performance: VAKER significantly outperforms baseline models across various metrics, including the lowest Fréchet Motion Distance (FMD) and Overlap, and the highest Maximum IoU (mIoU). For instance, VAKER achieved an FMD of 0.0103 and an Overlap of 0.4221, significantly better than other text-to-layout models. The failure rate was also substantially reduced, with approximately 3 to 6 times fewer instances of invalid layout generation.
  • Qualitative Improvements: VAKER generates high-quality layouts that accurately align with natural language prompts, producing aesthetically pleasing and precise typography. The generated animations are diverse and visually cohesive.
  • User Study Validation: A user study showed that participants preferred VAKER's results over a strong baseline (LayoutPrompter) for both layout quality (79.85% preference) and advertising effectiveness (76.17% preference).

Limitations and Future Work

  • Animation Complexity: The current ST-Representation simplifies animation by focusing on positional changes, omitting complex transformation parameters like rotation or scaling in its initial exploration.
  • Information Loss: There's a potential for partial information loss during the conversion from videos to STs.
  • Future Directions: The authors suggest exploring more complex animation patterns, dynamic visual effects, and improving information preservation during the ST conversion process. They also aim to extend the framework to handle more complex scene compositions and interactions.

Summary updated

Abstract

Despite the remarkable progress in text-to-video models, achieving precise control over text elements and animated graphics remains a significant challenge, especially in applications such as video advertisements. To address this limitation, we introduce Animated Layout Generation, a novel approach to extend static graphic layouts with temporal dynamics. We propose a Structured Text Representation for fine-grained video control through hierarchical visual elements. To demonstrate the effectiveness of our approach, we present VAKER (Video Ad maKER), a text-to-video advertisement generation pipeline that combines a three-stage generation process with Unstructured Text Reasoning for seamless integration with LLMs. VAKER fully automates video advertisement generation by incorporating dynamic layout trajectories for objects and graphics across specific video frames. Through extensive evaluations, we demonstrate that VAKER significantly outperforms existing methods in generating video advertisements. Project Page: this https URL
BibTeX
@misc{shin2025generating,
  author = {Yeonsang Shin and Jihwan Kim and Yumin Song and Kyungseung Lee and HyunHee Chung and Taeyoung Na},
  title = {{Generating Animated Layouts as Structured Text Representations}},
  year = {2025},
  eprint = {2505.00975},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.00975},
}
Figure