Scene Style Text Editing

Tonghua Su, Fuxiang Yang, Xiang Zhou, Donglin Di, Zhongjie Wang, Songze LiarXiv 2023 · DOI · arXiv

Abstract Traditional Scene Text Editing (STE) focuses primarily on content replacement, often struggling with fine-grained style control and background artifacts. This paper introduces the Scene Style Text Editing (SSTE) task and proposes QuadNet, a framework that decouples text content, style (rotation, font, color), and background. By leveraging a quadruple-module architecture and semantic latent space manipulation, QuadNet enables precise editing of text attributes in natural images while maintaining high visual fidelity.

Introduction

Editing text within natural scenes is a complex computer vision task with applications ranging from image translation to privacy masking. While existing STE methods can swap characters, they often exhibit "style rigidity"—an inability to adjust specific attributes like a sign's rotation or a font's weight—and frequently leave "ghosting" artifacts from the original text. To address these limitations, researchers from the Harbin Institute of Technology and Tsinghua University propose the Scene Style Text Editing (SSTE) task, treating text style as a manipulatable latent space.

The QuadNet Framework

The core of the proposed solution is QuadNet, which decomposes the editing process into four functional components to ensure clean backgrounds and stylized foregrounds:

  1. Background Inpainting Module: Utilizing a pre-trained LaMa model, this module identifies and erases the original text, recovering the underlying texture to provide a clean canvas.
  2. Style Encoder: Extracts a style embedding ($z$) from the source image to capture the visual essence of the original text.
  3. Content Encoder: Processes the target text string into a usable representation.
  4. Fusion Generator: The central integration unit. It employs StyleMapNet to map the style vector into hierarchical "layer vectors" ($w^i$), which are then injected into a feature pyramid via AdaIN (Adaptive Instance Normalization) to render the final stylized text.
The QuadNet architecture, featuring the Background Inpainting Module and the Foreground Style Editing Module (comprising Content/Style Encoders and a Fusion Generator).

Semantic Latent Editing

A key innovation of QuadNet is its ability to perform "semantic discovery" within the latent space. By interpolating between different layer vectors ($w^i$), the model can independently control specific attributes. Empirical findings suggest that $w^0$ primarily governs rotation, $w^{1,2,3}$ influence font characteristics, and $w^4$ dictates color. This decoupling allows for smooth, linear transitions across styles, such as rotating text from -10° to +10° or shifting from a serif to a sans-serif font without affecting other properties.

Qualitative results demonstrating QuadNet's ability to independently edit rotation, font, and color while preserving the background.

Experimental Results

QuadNet was evaluated on both synthetic and real-world datasets (including SROIE, COCO, and ICDAR). Because real-world data lacks paired ground truth, the authors utilized a shared-weight training approach and a pre-trained text recognizer to ensure legibility.

Quantitatively, QuadNet significantly outperforms previous state-of-the-art models like SRNet and Palette. On real-world benchmarks, QuadNet achieved a text recognition accuracy of 0.887 (vs. SRNet's 0.423) and superior image quality metrics, including a lower FID (37.32) and LPIPS (0.21). Qualitatively, the model produces crisp, artifact-free text that integrates seamlessly into the original scene's geometry and lighting.

Conclusion

QuadNet represents a significant step forward in scene text manipulation by moving beyond simple character replacement. By decoupling style from content and background, it offers a level of granular control previously unavailable in STE. Future work may focus on refining the inpainting process for more complex textures and extending geometric control to non-linear transformations like perspective warping on curved surfaces.

Summary updated

Abstract

In this work, we propose a task called "Scene Style Text Editing (SSTE)", changing the text content as well as the text style of the source image while keeping the original text scene. Existing methods neglect to fine-grained adjust the style of the foreground text, such as its rotation angle, color, and font type. To tackle this task, we propose a quadruple framework named "QuadNet" to embed and adjust foreground text styles in the latent feature space. Specifically, QuadNet consists of four parts, namely background inpainting, style encoder, content encoder, and fusion generator. The background inpainting erases the source text content and recovers the appropriate background with a highly authentic texture. The style encoder extracts the style embedding of the foreground text. The content encoder provides target text representations in the latent feature space to implement the content edits. The fusion generator combines the information yielded from the mentioned parts and generates the rendered text images. Practically, our method is capable of performing promisingly on real-world datasets with merely string-level annotation. To the best of our knowledge, our work is the first to finely manipulate the foreground text content and style by deeply semantic editing in the latent feature space. Extensive experiments demonstrate that QuadNet has the ability to generate photo-realistic foreground text and avoid source text shadows in real-world scenes when editing text content.
BibTeX
@misc{su2023scene,
  author = {Tonghua Su and Fuxiang Yang and Xiang Zhou and Donglin Di and Zhongjie Wang and Songze Li},
  title = {{Scene Style Text Editing}},
  year = {2023},
  eprint = {2304.10097},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.10097},
}
Figure