TextGround4M: A Prompt-Aligned Dataset for Layout-Aware Text Rendering

Dongxing Mao, Yilin Wang, Linjie Li, Zhengyuan Yang, Alex Jinpeng Wang

TL;DR

TextGround4M introduces a massive dataset of 4.1 million prompt-aligned, span-level annotated image pairs to solve the problem of inaccurate text rendering and placement in text-to-image (T2I) generation. Alongside the dataset, the authors propose a lightweight, training-time-only token appending strategy that significantly improves spatial text accuracy and prompt fidelity in autoregressive models without adding any inference overhead.

What Problem Does This Paper Address?

State-of-the-art text-to-image models (like DALL·E 3 or Flux) generate stunningly realistic images, but they frequently stumble when asked to render specific text in precise locations. This issue becomes particularly glaring in complex, multi-span layouts like posters, digital advertisements, or infographics.

The core bottleneck isn't just model capacity; it's data misalignment. Existing text-heavy datasets provide Optical Character Recognition (OCR) labels at the word or line level but fail to connect these regions back to the user prompt. As a result, models learn that text exists in images, but not what text goes where based on the prompt instructions.

Comparison of existing datasets and TextGround4M

Methods

The authors propose a lightweight training strategy tailored for autoregressive (AR) T2I models called Layout-posting (or Post-Image supervision).

  • Token Sequence Construction: During training, the target sequence is modified to append structured text and layout tokens after the standard image tokens. Each text region required by the prompt is tokenized as a pair: [TextTokens(text_content, bounding_box)], where coordinates are normalized to integers in the [0, 512] range.
  • Decoupled Loss: The training objective splits the loss into image generation ($L_{img}$) and text/layout prediction ($L_{text}$). This allows the model to optimize both visual realism and textual grounding separately.
  • Inference Simplicity: Crucially, at inference time, these extra layout tokens are completely omitted. The model generates images normally from text prompts alone, meaning zero architectural modifications or extra computational costs at test time.

Datasets

  • TextGround4M: A newly introduced dataset containing 4.1 million high-quality image-text pairs. It was built using a hybrid pipeline: aggregating 8 million public images and mining 3.7 million text-rich images from CommonCrawl via GPT-4o. These were annotated using Qwen2.5-VL for captioning and PaddleOCR for spatial text extraction, followed by rigorous multi-stage filtering to ensure semantic alignment.
  • TextGround-Bench: A new difficulty-aware evaluation benchmark stratified into Easy, Medium, and Hard levels based on the number of text spans ($N_{box}$) and maximum span token length ($W_{max}$).
Overview of the dataset construction pipeline

Key Insights and Ideas

  • The Power of Spatial Supervision: Bounding box tokens provide a much stronger learning signal for text placement than text tokens alone. However, combining both text and spatial coordinates yields the most robust results as layout complexity scales up.
  • Post-Image Ordering Wins: Appending layout tokens after image tokens (Post-Image) provides a clean, inference-consistent design while maintaining stable performance across all difficulty levels compared to putting them before the image tokens (Pre-Image).
  • Prompt-Grounded vs. Prompt-Agnostic: Traditional datasets label all visible text, including incidental background text. TextGround4M focuses strictly on text semantically entailed by the prompt, forcing the model to internalize user intent rather than just copying random text patterns.

Key Results

Fine-tuning a Janus Pro 1B model on TextGround4M yielded massive improvements across the board:

  • Prompt Coverage (PC): On the "Hard" benchmark split, the fine-tuned model maintained robust text rendering performance, resisting the sharp quality drops that plague other open-source baselines like TextDiffuser-2.
  • Layout Accuracy: The model demonstrated significantly better alignment between prompt instructions and image content, achieving accurate and well-positioned text rendering across diverse layouts without requiring explicit layout hints at test time.

Limitations and Future Work

While TextGround4M provides excellent spatial and textual annotations, it currently lacks fine-grained typographic details such as font styles, sizes, and colors. The authors plan to enrich the dataset with these stylistic attributes in future work to better support high-fidelity and highly stylized T2I generation tasks.

Summary updated

Abstract

Despite recent advances in text-to-image generation, models still struggle to accurately render prompt-specified text with correct spatial layout -- especially in multi-span, structured settings. This challenge is driven not only by the lack of datasets that align prompts with the exact text and layout expected in the image, but also by the absence of effective metrics for evaluating layout quality. To address these issues, we introduce TextGround4M, a large-scale dataset of over 4 million prompt-image pairs, each annotated with span-level text grounded in the prompt and corresponding bounding boxes. This enables fine-grained supervision for layout-aware, prompt-grounded text rendering. Building on this, we propose a lightweight training strategy for autoregressive T2I models that appends layout-aware span tokens during training, without altering model architecture or inference behavior. We further construct a benchmark with stratified layout complexity to evaluate both open-source and proprietary models in a zero-shot setting. In addition, we introduce two layout-aware metrics to address the long-standing lack of spatial evaluation in text rendering. Our results show that models trained on TextGround4M outperform strong baselines in text fidelity, spatial accuracy, and prompt consistency, highlighting the importance of fine-grained layout supervision for grounded T2I generation.
BibTeX
@inproceedings{mao2026textground,
  author = {Dongxing Mao and Yilin Wang and Linjie Li and Zhengyuan Yang and Alex Jinpeng Wang},
  title = {{TextGround4M: A Prompt-Aligned Dataset for Layout-Aware Text Rendering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i10.37736},
}

Loading related content...

Figure