Abstract
Scene text erasing is a task of removing text from natural scene images, which has been gaining attention in recent years. The main motivation is to conceal private information such as license plate numbers and house nameplates. In this work, we propose a method for scene text erasing that approaches the problem as a general inpainting task. In contrast to previous methods, which require pairs of original images containing text and images from which the text has been removed, our method does not need corresponding image pairs for training. We use a separately trained scene text detector and an inpainting network. The scene text detector predicts segmentation maps of text instances which are then used as masks for the inpainting network. The network for inpainting, trained on a large-scale image dataset, fills in masked out regions in an input image and generates a final image in which the original text is no longer present. The results show that our method is able to successfully remove text and fill in the created holes to produce natural-looking images.
Introduction
Scene text erasing is crucial for privacy preservation and photo editing. However, training models for this task is notoriously difficult due to the requirement for massive datasets of image pairs: one with text and an identical one where the text has been perfectly removed. Creating these pairs for real-world photos is labor-intensive, and synthetic data often results in a "domain gap" where models struggle with real-world lighting and textures. Researchers from the University of Tokyo have developed a method to erase text from natural scene images without needing these expensive "before and after" image pairs, using a "weakly supervised" approach.
Methodology: A Two-Stage Pipeline
The proposed method decouples text erasing into two specialized tasks:
- Text Detection (The Masker): The model uses a pre-trained scene text detector, PSENet, to generate a precise segmentation map of the text's location. Using segmentation instead of simple bounding boxes creates a tighter mask, reducing the area the inpainting network must "hallucinate."
- Image Inpainting (The Painter): The masked image is passed to a Generative Adversarial Network (GAN) designed for pluralistic image completion. This network, trained on millions of general images, fills in the holes by predicting what should be behind the missing regions.
This approach is considered "weakly supervised" because the system does not need to see a "clean" version of a specific text-heavy image during training; it only needs to know where text is and what natural scenes generally look like.
Experimental Setup
The authors leveraged several large-scale datasets to train the individual components:
- Detection: MLT 2017 and ICDAR 2015.
- Inpainting: Places2 (1.8 million images), Paris Street View, and ImageNet-1k.
- Evaluation: The ICDAR 2013 dataset and real-world benchmarks like EnsNet were used to test erasure effectiveness and visual quality.
Results and Performance
The model was evaluated on its ability to "fool" other text detectors and the visual quality of the resulting images. On the ICDAR 2013 benchmark, the model reduced text detection recall to as low as 0.64%, effectively removing almost all detectable text.
Qualitatively, the model proved particularly effective at blending text into complex backgrounds like brick walls or wooden signs, where simple blurring would be obvious. It achieved a Structural Similarity (SSIM) score of 92.73 on real-world datasets, performing competitively with fully supervised models like EnsNet.
Conclusion and Future Work
By separating detection from filling, the authors leverage massive general-purpose datasets, far larger than any existing text-erasing dataset. While highly effective, the method's primary limitation is its dependence on the detector; if the detector misses text, the inpainter cannot fix it. Future work may involve end-to-end training to further harmonize the mask generation and inpainting processes.
Summary updated