Compressing Colour Images with Joint Inpainting and Prediction

Rahul Mohideen Kaja Mohideen, Pascal Peter, Tobias Alt, Joachim Weickert, Alexander ScheerarXiv 2020 · DOI · arXiv

Abstract

Inpainting-based codecs store sparse, quantised pixel data directly and decode by interpolating the discarded image parts. This interpolation can be used simultaneously for efficient coding by predicting pixel data to be stored. Such joint inpainting and prediction approaches yield good results with simple components such as regular grids and Shepard interpolation on grey value images, but they lack a dedicated mode for colour images. Therefore, we evaluate different approaches for inpainting-based colour compression. Inpainting operators are able to reconstruct a large range of colours from a small colour palette of the known pixels. We exploit this with a luma preference mode which uses higher sparsity in YCbCr colour channels than in the brightness channel. Furthermore, we propose the first full vector quantisation mode for an inpainting-based codec that stores only a small codebook of colours. Our experiments reveal that both colour extensions yield significant improvements.

Introduction

Traditional image compression, such as JPEG, relies on mathematical transforms to reduce file sizes. A compelling alternative is found in inpainting-based codecs. These systems function by discarding the vast majority of image data, retaining only a sparse "mask" of pixels. During decompression, the decoder employs interpolation algorithms to "fill in the blanks" and reconstruct the original scene.

While highly effective for grayscale imagery, these methods historically struggle with color. Applying grayscale techniques to Red, Green, and Blue (RGB) channels independently is inefficient because it fails to account for the human eye's sensitivity to brightness (luma) over color (chroma). Furthermore, standard inpainting does not exploit the fact that a rich spectrum of hues can often be reconstructed from a very limited set of "seed" colors.

Methodology: LP and VQ Modes

This research builds upon the Regular Grid Joint Inpainting and Prediction (RJIP) codec. RJIP stores pixels on a simple grid and utilizes Shepard interpolation—a weighted average based on distance—to reconstruct the image. To optimize this for color, two novel modes were introduced:

  1. Luma Preference (LP) Mode: The image is converted from RGB to the YCbCr color space. The "Y" (brightness) channel is allocated a significantly higher data budget (a denser pixel grid) than the "Cb" and "Cr" (chroma) channels. This ensures that sharp edges and structural details are preserved while color is smoothly interpolated across the scene.
  2. Vector Quantisation (VQ) Mode: Instead of storing raw color values for every mask pixel, the encoder employs k-means clustering to generate a small, image-specific color palette (codebook). Each stored pixel then only needs to save an index pointing to a color in this palette, drastically reducing the storage requirement per pixel.

Additionally, the researchers implemented Advanced Tonal Optimization. By replacing iterative "random walk" methods with a direct mathematical closed-form solution, the codec can analytically determine the optimal color values for mask pixels to minimize reconstruction error, significantly accelerating the encoding process.

Experimental Results

The proposed methods were evaluated using the Kodak image database, a standard benchmark for high-quality image processing. The results demonstrated that inpainting acts as a powerful "co-domain" interpolator, generating complex gradients and hues from very sparse data.

The reconstruction pipeline for the kodim07 image: (a) the original image and its color distribution, (b) the highly sparse inpainting mask, and (c) the final reconstruction showing high fidelity to the original.

Key performance metrics include:

  • Error Reduction: Both new modes consistently outperformed standard RGB scalar inpainting. In low-texture images (e.g., clear skies), the VQ mode reduced the Mean Squared Error (MSE) by up to 54%.
  • Efficiency: The VQ mode proved to be computationally superior, achieving a runtime reduction of up to 50% compared to the LP mode, as it avoids the need to search for an optimal luma-to-chroma data balance.
  • Visual Quality: The qualitative results show a marked reduction in "blockiness" and color bleeding, common artifacts in simpler inpainting codecs.

Conclusion and Future Work

The study confirms that joint inpainting and prediction can be highly effective for color images when leveraging luma preference or vector quantisation. While successful, challenges remain in predicting palette indices (labels) for VQ mode due to their non-mathematical relationship with spatial distance. Future research may explore extending these techniques to hyperspectral imaging, where the high dimensionality of data makes the benefits of vector quantisation even more pronounced.

Summary updated

Abstract

Inpainting-based codecs store sparse, quantised pixel data directly and decode by interpolating the discarded image parts. This interpolation can be used simultaneously for efficient coding by predicting pixel data to be stored. Such joint inpainting and prediction approaches yield good results with simple components such as regular grids and Shepard interpolation on grey value images, but they lack a dedicated mode for colour images. Therefore, we evaluate different approaches for inpainting-based colour compression. Inpainting operators are able to reconstruct a large range of colours from a small colour palette of the known pixels. We exploit this with a luma preference mode which uses higher sparsity in YCbCr colour channels than in the brightness channel. Furthermore, we propose the first full vector quantisation mode for an inpainting-based codec that stores only a small codebook of colours. Our experiments reveal that both colour extensions yield significant improvements.
BibTeX
@misc{mohideen2020compressing,
  author = {Rahul Mohideen Kaja Mohideen and Pascal Peter and Tobias Alt and Joachim Weickert and Alexander Scheer},
  title = {{Compressing Colour Images with Joint Inpainting and Prediction}},
  year = {2020},
  eprint = {2010.09866},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.09866},
}
Figure