Abstract
In this paper, we introduce a new dataset of room interior pictures with overlaying and scene text, totaling 4,836 annotated images across 25 product categories. We provide details on the collection and annotation process of our dataset and analyze its statistics. Furthermore, we propose a baseline method for overlaying text detection that leverages the character region-aware text detection framework to guide a classification model. We validate our approach and demonstrate its efficiency in terms of binary classification metrics, reaching a final performance of 0.95 F1 score, with false positive and false negative rates of 0.02 and 0.06, respectively.
Introduction
In the domains of e-commerce and social media, text detection is a fundamental task. However, standard Optical Character Recognition (OCR) tools typically fail to distinguish between "scene text"—text naturally occurring in an environment, such as a book title or a clock face—and "overlaying text"—digitally added content like watermarks, promotional banners, or spam.
This distinction is critical for fraud detection and content moderation. For example, a photo of a living room containing a book titled "Design" is legitimate, whereas the same photo with a digital "50% OFF" banner might violate platform policies. Existing models often struggle in "organic" environments like home interiors, where the high density of natural scene text leads to excessive false-positive rates for overlay detection. This paper addresses this gap by introducing a specialized dataset and a robust detection methodology.
The Rooms-with-Text Dataset
The primary contribution of this work is the Rooms-with-Text dataset, designed to benchmark the separation of digital overlays from natural scene text in complex environments.
- Scale and Diversity: The dataset contains 4,836 annotated images spanning 25 product categories, primarily within the Home and Furniture sectors.
- Content: Images feature real-world e-commerce scenarios with varying combinations of scene text and overlays.
- Annotation: Images are categorized into four labels: "Overlaying," "Organic," "Both," or "None." For binary classification tasks, these are collapsed into Positive (contains overlay) and Negative (no overlay) classes.
- Data Splits: The dataset is divided into 3,627 images for training and 1,209 for validation.
To ensure the dataset was high-quality and relevant, the authors employed a heuristic filtering rule based on initial text region scores to exclude images likely to contain no text, focusing manual annotation efforts on challenging cases.
Methodology
We propose a two-stage pipeline that treats overlay detection as a binary classification task, utilizing spatial attention to improve accuracy.
- Text Localization: The system uses the CRAFT (Character Region Awareness for Text) framework as a backbone. Rather than simple bounding boxes, CRAFT provides "region scores" that represent the probability of text presence at the pixel level.
- Mask Generation: These region scores are used to create a soft proposal mask. This mask is upsampled and multiplied element-wise with the original input image.
- Guided Classification: The resulting masked image, which effectively "highlights" text regions while suppressing background noise, is fed into a ResNet-18 classifier. This forces the model to focus on the visual signatures of digital overlays—such as perfect alignment, specific digital fonts, and a lack of perspective distortion—compared to the varied appearance of scene text.
Key Results
The proposed CRAFT-masked ResNet significantly outperformed both a vanilla ResNet-18 and existing masked architectures like SplitNet. By focusing the model's attention on text regions, we achieved a substantial boost in F1 score and a dramatic reduction in the False Positive Rate (FPR).
| Method | AUC | Precision | Recall | F1 Score | FPR | FNR |
|---|---|---|---|---|---|---|
| CRAFT-masked (Proposed) | 0.99 | 0.97 | 0.94 | 0.95 | 0.02 | 0.06 |
| Vanilla ResNet-18 | 0.93 | 0.82 | 0.80 | 0.81 | 0.11 | 0.20 |
| SplitNet-masked | 0.76 | 0.62 | 0.57 | 0.59 | 0.21 | 0.43 |
The low FPR (0.02) is particularly noteworthy for industrial applications, as it minimizes the risk of incorrectly flagging legitimate user content in automated moderation systems.
Conclusion and Future Work
This work demonstrates that spatial guidance via text localization masks is a highly effective strategy for distinguishing digital overlays from organic scene text. While the model is robust, it faces challenges with extremely small fonts or text at the image periphery. Future research could extend this work toward image restoration, where detected overlays are automatically removed and the underlying scene is inpainted to restore the image's original state. Additionally, expanding the dataset to include more diverse "organic" environments beyond home interiors will further enhance model generalization.
Summary updated