Abstract Traditional Scene Text Editing (STE) focuses primarily on content replacement, often struggling with fine-grained style control and background artifacts. This paper introduces the Scene Style Text Editing (SSTE) task and proposes QuadNet, a framework that decouples text content, style (rotation, font, color), and background. By leveraging a quadruple-module architecture and semantic latent space manipulation, QuadNet enables precise editing of text attributes in natural images while maintaining high visual fidelity.
Introduction
Editing text within natural scenes is a complex computer vision task with applications ranging from image translation to privacy masking. While existing STE methods can swap characters, they often exhibit "style rigidity"—an inability to adjust specific attributes like a sign's rotation or a font's weight—and frequently leave "ghosting" artifacts from the original text. To address these limitations, researchers from the Harbin Institute of Technology and Tsinghua University propose the Scene Style Text Editing (SSTE) task, treating text style as a manipulatable latent space.
The QuadNet Framework
The core of the proposed solution is QuadNet, which decomposes the editing process into four functional components to ensure clean backgrounds and stylized foregrounds:
- Background Inpainting Module: Utilizing a pre-trained LaMa model, this module identifies and erases the original text, recovering the underlying texture to provide a clean canvas.
- Style Encoder: Extracts a style embedding ($z$) from the source image to capture the visual essence of the original text.
- Content Encoder: Processes the target text string into a usable representation.
- Fusion Generator: The central integration unit. It employs StyleMapNet to map the style vector into hierarchical "layer vectors" ($w^i$), which are then injected into a feature pyramid via AdaIN (Adaptive Instance Normalization) to render the final stylized text.
Semantic Latent Editing
A key innovation of QuadNet is its ability to perform "semantic discovery" within the latent space. By interpolating between different layer vectors ($w^i$), the model can independently control specific attributes. Empirical findings suggest that $w^0$ primarily governs rotation, $w^{1,2,3}$ influence font characteristics, and $w^4$ dictates color. This decoupling allows for smooth, linear transitions across styles, such as rotating text from -10° to +10° or shifting from a serif to a sans-serif font without affecting other properties.
Experimental Results
QuadNet was evaluated on both synthetic and real-world datasets (including SROIE, COCO, and ICDAR). Because real-world data lacks paired ground truth, the authors utilized a shared-weight training approach and a pre-trained text recognizer to ensure legibility.
Quantitatively, QuadNet significantly outperforms previous state-of-the-art models like SRNet and Palette. On real-world benchmarks, QuadNet achieved a text recognition accuracy of 0.887 (vs. SRNet's 0.423) and superior image quality metrics, including a lower FID (37.32) and LPIPS (0.21). Qualitatively, the model produces crisp, artifact-free text that integrates seamlessly into the original scene's geometry and lighting.
Conclusion
QuadNet represents a significant step forward in scene text manipulation by moving beyond simple character replacement. By decoupling style from content and background, it offers a level of granular control previously unavailable in STE. Future work may focus on refining the inpainting process for more complex textures and extending geometric control to non-linear transformations like perspective warping on curved surfaces.
Summary updated