GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editing

Tong Wang, Ting Liu, Xiaochao Qu, Chengjing Wu, Luoqi Liu, Xiaolin Hu

TL;DR

GlyphMastero is a novel, trainable glyph encoder designed to bring stroke-level precision to diffusion-based scene text editing. By explicitly modeling the hierarchical relationships between individual characters and entire text lines, it prevents the character distortion common in complex scripts like Chinese. The method achieves a massive 18.02% improvement in sentence accuracy and slashes text-region style inconsistency (FID) by over 53% compared to current state-of-the-art baselines.

What Problem Does This Paper Address?

Scene text editing—modifying text directly within an image while preserving the original font, lighting, and background style—is a notoriously tricky task. While latent diffusion models have advanced image editing significantly, they frequently stumble when generating text. Characters often turn out warped, blurry, or entirely unreadable.

This issue is amplified when dealing with logographic writing systems like Chinese, where characters are composed of intricate, highly specific stroke patterns and spatial arrangements. Current state-of-the-art methods try to solve this by feeding features from pretrained Optical Character Recognition (OCR) models into the diffusion process. However, they treat text features as flat representations, failing to capture the hierarchical nature of text—specifically, how individual local strokes build characters, and how those characters interact within a global text line.

Methods

To bridge this structural gap, the authors introduce GlyphMastero, a specialized dual-stream glyph encoder that plugs into an inpainting-based latent diffusion backbone (Stable Diffusion 2.1).

Figure 3. Complete model architecture of GlyphMastero. A specialized glyph encoder that introduces stroke-level precise control to the latent diffusion model for scene text editing.

The core architecture relies on three main innovations:

  1. Dual-Stream Feature Extraction: The model processes text conditions via two parallel paths using a pretrained PaddleOCR-v4 model. The local-level stream processes rendered images of individual characters, while the global-level stream processes the rendered image of the entire text line.
  2. Feature Pyramid Network (FPN): In the global stream, an FPN is introduced to fuse multi-scale OCR backbone features. This allows the model to retain high-resolution, fine-grained stroke details from shallow layers alongside the rich semantic context from deeper layers.
  3. Glyph Attention Module: To make these two streams talk to each other, the authors designed a novel cross-attention mechanism. It uses Rotary Positional Embeddings (RoPE) to map the interactions between local character features (acting as queries) and global line features (acting as keys and values), ensuring precise spatial and structural alignment.
Figure 4. Glyph Attention Module

Datasets

The model was trained and evaluated using several key benchmarks:

  • AnyWord-3M: A massive dataset containing roughly 3.5 million images with line-level annotations, used for large-scale training.
  • AnyText-Eval: Used for quantitative evaluation, consisting of 2,000 images split into 4,181 English and 2,092 Chinese text-position pairs.
  • TextCtrl's ScenePair: A dataset of 1,285 test cases used specifically to validate English-only performance.
  • Curated Stylistic Dataset: A newly compiled evaluation set of 80 highly challenging, highly stylistic images (120 text-position pairs) designed to test the limits of text style preservation and content replacement.

Key Insights and Ideas

  • Text is Hierarchical, Not Flat: The defining insight of this paper is that text generation requires explicit multi-level awareness. Treating an OCR feature map as a uniform conditioning signal throws away crucial structural boundaries. Explicitly separating and then cross-attending character-level and line-level features acts as a strong regularizer against character distortion.
  • Shallow Features Matter for Text: Standard deep learning architectures prioritize deep, semantic features. However, for text rendering, the fine-grained geometric details found in shallower layers are critical. The FPN integration proved vital for capturing these subtle stroke boundaries.
  • The CFG Readability Trade-off: The authors uncovered a crucial trade-off regarding Classifier-Free Guidance (CFG) in scene text editing. Higher CFG scales enforce stronger glyph control (making text clearer and thicker), but this comes at the expense of style preservation. Lower CFG scales match the original background style better but risk generating inaccurate text.
Visualizing the trade-off of different Classifier-Free Guidance (CFG) scales on text readability and style.

Key Results

GlyphMastero significantly outperforms previous multi-lingual baselines like DiffUTE and AnyText across both text accuracy and style consistency metrics.

  • Multi-lingual Performance: On the AnyText-Eval benchmark, GlyphMastero achieved an overall sentence accuracy improvement of 18.02% over the baselines, while simultaneously reducing the Character Error Rate (CER).
  • Fidelity and Style: The text-region Fréchet Inception Distance (FID) was reduced by 53.28% on average, indicating that the edited text blends seamlessly into the original image environment without looking like a cheap digital overlay.
Qualitative comparison showing GlyphMastero's superior style preservation and character accuracy compared to DiffUTE and AnyText.

Limitations and Future Work

Despite its strengths, GlyphMastero still faces challenges when generating exceptionally long text strings. When the target text length is significantly shorter than the selected bounding box area, the model occasionally struggles with spatial distribution, leading to irregularly sized characters or repetitive patterns.

The authors attribute this to the base diffusion model's constraints and the fact that training data was capped at a $512 \times 512$ resolution. Future work will focus on scaling the framework to higher resolutions and integrating it with more robust, next-generation latent diffusion base models.

Summary updated

Abstract

Scene text editing, a subfield of image editing, requires modifying texts in images while preserving style consistency and visual coherence with the surrounding environment. While diffusion-based methods have shown promise in text generation, they still struggle to produce high-quality results. These methods often generate distorted or unrecognizable characters, particularly when dealing with complex characters like Chinese. In such systems, characters are composed of intricate stroke patterns and spatial relationships that must be precisely maintained. We present GlyphMastero, a specialized glyph encoder designed to guide the latent diffusion model for generating texts with stroke-level precision. Our key insight is that existing methods, despite using pretrained OCR models for feature extraction, fail to capture the hierarchical nature of text structures - from individual strokes to stroke-level interactions to overall character-level structure. To address this, our glyph encoder explicitly models and captures the cross-level interactions between local-level individual characters and global-level text lines through our novel glyph attention module. Meanwhile, our model implements a feature pyramid network to fuse the multi-scale OCR backbone features at the global-level. Through these cross-level and multi-scale fusions, we obtain more detailed glyph-aware guidance, enabling precise control over the scene text generation process. Our method achieves an 18.02\% improvement in sentence accuracy over the state-of-the-art multi-lingual scene text editing baseline, while simultaneously reducing the text-region Fréchet inception distance by 53.28\%.
BibTeX
@inproceedings{wang2025glyphmastero,
  author = {Tong Wang and Ting Liu and Xiaochao Qu and Chengjing Wu and Luoqi Liu and Xiaolin Hu},
  title = {{GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editing}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02656},
}
Figure