Text extraction from texture images using masked signal decomposition

Shervin Minaee, Yao Wang

Abstract

Text extraction is an important problem in image processing with applications from optical character recognition to autonomous driving. Most of the traditional text segmentation algorithms consider separating text from a simple background (which usually has a different color from texts). In this work we consider separating texts from a textured background, that has similar color to texts. We look at this problem from a signal decomposition perspective, and consider a more realistic scenario where signal components are overlaid on top of each other (instead of adding together). When the signals are overlaid, to separate signal components, we need to find a binary mask which shows the support of each component. Because directly solving the binary mask is intractable, we relax this problem to the approximated continuous problem, and solve it by alternating optimization method. We show that the proposed algorithm achieves significantly better results than other recent works on several challenging images.

Introduction

Text extraction is a foundational step for OCR, autonomous driving, and document analysis. However, most existing algorithms struggle when the background is not a simple, solid color. When a background has a complex texture or a color profile nearly identical to the text, traditional clustering (like k-means) or simple thresholding fails to produce clean results.

Text overlaid on a complex grayscale textured background, illustrating the difficulty of separation when color distributions overlap.

The authors identify a specific mathematical gap: most signal decomposition models assume an additive model ($x = x_1 + x_2$). In reality, text is an overlay; a pixel is either text or background, not a sum of both. This paper tackles the challenge of separating these components when they are non-linearly combined via a binary mask.

Methods: Masked Signal Decomposition

The core of the paper is the "Masked Signal Decomposition" model. They represent an image $x$ as: $$x = (1 - w) \circ x_1 + w \circ x_2$$ Where $x_1$ is the background, $x_2$ is the foreground text, $w$ is a binary mask (1 for text, 0 for background), and $\circ$ denotes the element-wise product.

To solve this, the authors make two key assumptions based on signal processing priors:

  1. Subspace Sparsity: Both the background and text can be represented sparsely in specific mathematical "subspaces." They use a Discrete Cosine Transform (DCT) subspace for the background (good for smooth/textured regions) and a Hadamard subspace for the text (good for sharp edges and strokes).
  2. Mask Sparsity: Text usually occupies a smaller portion of the image than the background, so the mask $w$ should be sparse.

Because finding a binary mask is computationally intractable, they relax the problem, allowing $w$ to be any continuous value between 0 and 1 during optimization. They solve the resulting equations using an alternating optimization method, where they iteratively update the background coefficients, the text coefficients, and the mask until the solution converges. A key discovery was that binarizing the mask only at the final step, rather than during iterations, prevents the algorithm from getting stuck in poor local minima.

Experimental Results

The authors evaluated their method on a dataset of more than 300 image blocks (64x64 pixels). These images were manually generated by overlaying text onto various complex, textured backgrounds to create a challenging ground-truth benchmark.

Comparison of text extraction results across different textured backgrounds using the proposed masked signal decomposition method.

The proposed algorithm significantly outperformed several industry-standard and academic baselines, including hierarchical clustering and sparse decomposition with Total Variation (TV).

Algorithm Precision Recall F1 Score
SPEC 67% 77% 71.6%
Hierarchical Clustering 66.5% 92% 77.2%
Sparse Dec. with TV 71% 91.7% 80%
Proposed Algorithm 95% 92.5% 93.7%

Qualitatively, the method shows a remarkable ability to "clean" text from backgrounds that look almost identical to the text color, where other methods leave significant noise or "ghosting" artifacts.

Conclusion

By shifting from an additive to an overlay model, the authors provide a more physically accurate framework for text extraction. The use of specific subspaces (DCT and Hadamard) tailored to the characteristics of backgrounds and text strokes, combined with a continuous relaxation of the binary mask, allows for robust performance on highly textured images. Future work may involve learning these subspaces dynamically to adapt to a wider variety of document types.

Summary updated

Abstract

Text extraction is an important problem in image processing with applications from optical character recognition to autonomous driving. Most of the traditional text segmentation algorithms consider separating text from a simple background (which usually has a different color from texts). In this work we consider separating texts from a textured background, that has similar color to texts. We look at this problem from a signal decomposition perspective, and consider a more realistic scenario where signal components are overlaid on top of each other (instead of adding together). When the signals are overlaid, to separate signal components, we need to find a binary mask which shows the support of each component. Because directly solving the binary mask is intractable, we relax this problem to the approximated continuous problem, and solve it by alternating optimization method. We show that the proposed algorithm achieves significantly better results than other recent works on several challenging images.
BibTeX
@misc{minaee2017text,
  author = {Shervin Minaee and Yao Wang},
  title = {{Text extraction from texture images using masked signal decomposition}},
  year = {2017},
  eprint = {1706.04041},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.04041},
}
Figure