Screen content image segmentation using sparse-smooth decomposition

Shervin Minaee, AmirAli Abdolrashidi, Yao Wang

Abstract

Screen content images (SCIs), which combine computer-generated graphics, text, and natural backgrounds, present unique challenges for traditional image processing. This paper proposes a segmentation framework that separates the background and foreground using a sparse-smooth decomposition technique. By modeling the background as a smooth component and the foreground as a sparse component, the algorithm achieves superior performance in text and graphic extraction. Evaluated on HEVC test sequences, the method demonstrates a significant improvement in recall (0.95) compared to existing hierarchical clustering methods.

Introduction

Standard image compression standards like JPEG2000 or HEVC are primarily designed for natural photographic content characterized by smooth transitions. Screen Content Images (SCIs), however, contain sharp edges and high-contrast UI elements. Applying standard transform-domain coding to these sharp discontinuities often leads to energy leakage across coefficients, resulting in high bit-rates or visible "ringing" artifacts. To optimize compression and analysis, it is essential to segment SCIs into a smooth background layer and a sharp foreground layer.

A typical screen content image featuring a complex color gradient background and sharp wireframe foreground elements.

Methodology

The proposed approach utilizes a Sparse-Smooth Decomposition model. The core premise is that the background can be represented by a linear combination of low-frequency basis functions, while the foreground is "sparse," occupying a small fraction of the image but containing high-frequency information.

The Mathematical Model

Each image block is modeled as: $$\hat{F}(x, y) = \sum_{k=1}^{K} \alpha_k P_k(x, y) + S(x, y)$$ Where $P_k(x, y)$ represents the first 10 low-frequency 2D Discrete Cosine Transform (DCT) bases, and $S(x, y)$ is the sparse foreground component. The separation is achieved by solving a Basis Pursuit problem, minimizing the $L_1$-norm of both the coefficients ($\alpha$) and the sparse component ($s$) using the Alternating Direction Method of Multipliers (ADMM).

To ensure computational efficiency, the authors implement a tiered pipeline:

  1. Flat Block Check: Immediate categorization of uniform color blocks.
  2. Smooth Block Check: Least-squares fit for purely smooth blocks.
  3. Sparse Decomposition: Full ADMM optimization for complex blocks.

Experimental Results

The algorithm was tested on a dataset of 332 image blocks derived from HEVC test sequences. The proposed scheme was compared against Shape Primitive Extraction and Coding (SPEC) and Hierarchical k-means (DjVu).

The extracted foreground mask from the sample image, demonstrating the algorithm's ability to capture sharp edges despite background gradients.

Quantitative Performance

The quantitative results highlight a major breakthrough in recall:

Algorithm Precision Recall
SPEC 0.50 0.64
Hierarchical Clustering (DjVu) 0.64 0.69
Proposed Scheme 0.64 0.95

A recall of 0.95 indicates that the algorithm captures nearly all foreground details, which is critical for maintaining legibility in compressed screen content. Unlike color-based clustering, this smoothness-based approach successfully handles cases where foreground and background colors overlap by focusing on the nature of the signal rather than just color distance.

Conclusion

By leveraging the physical nature of image signals—smoothness for backgrounds and sparsity for foregrounds—this decomposition method provides a robust solution for SCI segmentation. While computationally more intensive than simple clustering, its high accuracy makes it a strong candidate for advanced video compression, text extraction, and potentially medical imaging applications.

Summary updated

Abstract

Sparse decomposition has been extensively used for different applications including signal compression and denoising and document analysis. In this paper, sparse decomposition is used for image segmentation. The proposed algorithm separates the background and foreground using a sparse-smooth decomposition technique such that the smooth and sparse components correspond to the background and foreground respectively. This algorithm is tested on several test images from HEVC test sequences and is shown to have superior performance over other methods, such as the hierarchical k-means clustering in DjVu. This segmentation algorithm can also be used for text extraction, video compression and medical image segmentation.
BibTeX
@misc{minaee2015screen,
  author = {Shervin Minaee and AmirAli Abdolrashidi and Yao Wang},
  title = {{Screen content image segmentation using sparse-smooth decomposition}},
  year = {2015},
  eprint = {1511.06911},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1511.06911},
}
Figure