De-rendering Stylized Texts

Wataru Shimoda, Daichi Haraguchi, Seiichi Uchida, Kota Yamaguchi

Editing text within a flattened image—such as a movie poster, advertisement, or web banner—has traditionally been a labor-intensive task for graphic designers. Conventional methods require manually cloning out existing text and attempting to match fonts, while modern AI-based "style transfer" tools often introduce blurry artifacts and lack precision.

A new research paper from CyberAgent and Kyushu University proposes a paradigm shift: De-rendering. Rather than treating text editing as a pixel-to-pixel translation problem, the authors treat it as a "vectorization" problem, decomposing a raster image into its underlying design parameters.

The De-rendering Framework

The core of this approach is a two-stage "Analysis-by-Synthesis" framework that converts stylized raster text into editable parameters like font, color, size, and effects (such as shadows and borders).

The complete de-rendering pipeline, showing the transition from a flattened input image to extracted parameters and subsequent edits.

The process involves three primary components:

  1. Feedforward Parser: Utilizing an architecture based on CharNet, this module detects text location, recognizes characters, and provides an initial estimate of styling attributes, including font category and colors.
  2. Feedback Refiner: To achieve high precision, the system employs differentiable rendering. It renders a new image based on the initial parameters and compares it to the original. Through backpropagation, the system "wiggles" these parameters—adjusting shadow offsets or warping character shapes—until the reconstruction perfectly matches the input.
  3. Background Inpainting: Simultaneously, a separate module (based on a modified MEDFE architecture) removes the original text and fills the resulting void, creating a clean canvas for the edited content.

Technical Innovations: Parameters Over Pixels

A significant challenge in de-rendering is that fonts are discrete entities; one cannot mathematically "slide" from one typeface to another. The researchers overcame this by using differentiable font approximation. They represent text as a weighted linear combination of pre-rendered alpha maps from a diverse library of fonts, allowing the optimizer to smoothly converge on the best-fitting typeface.

An example of the specific design parameters recovered by the de-rendering process, enabling precise, resolution-independent editing.

By moving the editing task into "parameter space," the output becomes naturally resolution-independent. Unlike GAN-based editors that are limited by training data resolution, this method produces results that can be scaled to any size without loss of quality, as the final output is rendered by a standard 2D graphics engine.

Key Results and Insights

The inclusion of a feedback loop proved critical. Quantitative evaluations on a Book Cover dataset showed that the refinement stage improved the Peak Signal-to-Noise Ratio (PSNR) from 29.22 to 30.02. Qualitatively, the method produces much sharper results than leading GAN-based models like SRNet, which often struggle with inconsistent edges on straight-lined fonts.

Ablation studies revealed that accurately capturing "Color" is the most vital factor for visual fidelity, followed closely by "Font" selection. This approach essentially redefines OCR, moving it beyond simple character recognition toward a complete understanding of visual intent.

Limitations and Future Outlook

While powerful, the current system has limitations. It is presently restricted to a vocabulary of three effects—fill, border, and shadow—and cannot yet handle complex textures (like "fire" text) or 3D extrusions. Additionally, the refinement process takes approximately one minute per word, which, while suitable for professional design tools, is not yet fast enough for real-time applications.

Future work aims to expand the library of supported effects and improve the speed of the differentiable rendering pipeline to handle more complex blending modes and custom, hand-drawn fonts.

Summary updated

Abstract

Editing raster text is a promising but challenging task. We propose to apply text vectorization for the task of raster text editing in display media, such as posters, web pages, or advertisements. In our approach, instead of applying image transformation or generation in the raster domain, we learn a text vectorization model to parse all the rendering parameters including text, location, size, font, style, effects, and hidden background, then utilize those parameters for reconstruction and any editing task. Our text vectorization takes advantage of differentiable text rendering to accurately reproduce the input raster text in a resolution-free parametric format. We show in the experiments that our approach can successfully parse text, styling, and background information in the unified model, and produces artifact-free text editing compared to a raster baseline.
BibTeX
@inproceedings{shimoda2021de,
  author = {Wataru Shimoda and Daichi Haraguchi and Seiichi Uchida and Kota Yamaguchi},
  title = {{De-rendering Stylized Texts}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00111},
}
Figure