TL;DR
TextGround4M introduces a massive dataset of 4.1 million prompt-aligned, span-level annotated image pairs to solve the problem of inaccurate text rendering and placement in text-to-image (T2I) generation. Alongside the dataset, the authors propose a lightweight, training-time-only token appending strategy that significantly improves spatial text accuracy and prompt fidelity in autoregressive models without adding any inference overhead.
What Problem Does This Paper Address?
State-of-the-art text-to-image models (like DALL·E 3 or Flux) generate stunningly realistic images, but they frequently stumble when asked to render specific text in precise locations. This issue becomes particularly glaring in complex, multi-span layouts like posters, digital advertisements, or infographics.
The core bottleneck isn't just model capacity; it's data misalignment. Existing text-heavy datasets provide Optical Character Recognition (OCR) labels at the word or line level but fail to connect these regions back to the user prompt. As a result, models learn that text exists in images, but not what text goes where based on the prompt instructions.
Methods
The authors propose a lightweight training strategy tailored for autoregressive (AR) T2I models called Layout-posting (or Post-Image supervision).
- Token Sequence Construction: During training, the target sequence is modified to append structured text and layout tokens after the standard image tokens. Each text region required by the prompt is tokenized as a pair:
[TextTokens(text_content, bounding_box)], where coordinates are normalized to integers in the[0, 512]range. - Decoupled Loss: The training objective splits the loss into image generation ($L_{img}$) and text/layout prediction ($L_{text}$). This allows the model to optimize both visual realism and textual grounding separately.
- Inference Simplicity: Crucially, at inference time, these extra layout tokens are completely omitted. The model generates images normally from text prompts alone, meaning zero architectural modifications or extra computational costs at test time.
Datasets
- TextGround4M: A newly introduced dataset containing 4.1 million high-quality image-text pairs. It was built using a hybrid pipeline: aggregating 8 million public images and mining 3.7 million text-rich images from CommonCrawl via GPT-4o. These were annotated using Qwen2.5-VL for captioning and PaddleOCR for spatial text extraction, followed by rigorous multi-stage filtering to ensure semantic alignment.
- TextGround-Bench: A new difficulty-aware evaluation benchmark stratified into Easy, Medium, and Hard levels based on the number of text spans ($N_{box}$) and maximum span token length ($W_{max}$).
Key Insights and Ideas
- The Power of Spatial Supervision: Bounding box tokens provide a much stronger learning signal for text placement than text tokens alone. However, combining both text and spatial coordinates yields the most robust results as layout complexity scales up.
- Post-Image Ordering Wins: Appending layout tokens after image tokens (Post-Image) provides a clean, inference-consistent design while maintaining stable performance across all difficulty levels compared to putting them before the image tokens (Pre-Image).
- Prompt-Grounded vs. Prompt-Agnostic: Traditional datasets label all visible text, including incidental background text. TextGround4M focuses strictly on text semantically entailed by the prompt, forcing the model to internalize user intent rather than just copying random text patterns.
Key Results
Fine-tuning a Janus Pro 1B model on TextGround4M yielded massive improvements across the board:
- Prompt Coverage (PC): On the "Hard" benchmark split, the fine-tuned model maintained robust text rendering performance, resisting the sharp quality drops that plague other open-source baselines like TextDiffuser-2.
- Layout Accuracy: The model demonstrated significantly better alignment between prompt instructions and image content, achieving accurate and well-positioned text rendering across diverse layouts without requiring explicit layout hints at test time.
Limitations and Future Work
While TextGround4M provides excellent spatial and textual annotations, it currently lacks fine-grained typographic details such as font styles, sizes, and colors. The authors plan to enrich the dataset with these stylistic attributes in future work to better support high-fidelity and highly stylized T2I generation tasks.
Summary updated