Object-Based Image Coding: A Learning-Driven Revisit

Qi Xia, Haojie Liu, Zhan Ma

Abstract

The Object-Based Image Coding (OBIC) that was extensively studied about two decades ago, promised a vast application perspective for both ultra-low bitrate communication and high-level semantical content understanding, but it had rarely been used due to the inefficient compact representation of object with arbitrary shape. A fundamental issue behind is how to efficiently process the arbitrary-shaped objects at a fine granularity (e.g., feature element or pixel wise). To attack this, we have proposed to apply the element-wise masking and compression by devising an object segmentation network for image layer decomposition, and parallel convolution-based neural image compression networks to process masked foreground objects and background scene separately. All components are optimized in an end-to-end learning framework to intelligently weigh their (e.g., object and background) contributions for visually pleasant reconstruction. We have conducted comprehensive experiments to evaluate the performance on PASCAL VOC dataset at a very low bitrate scenario (e.g., ≤ 0.1 bits per pixel - bpp) which have demonstrated noticeable subjective quality improvement compared with JPEG2K, HEVC-based BPG and another learned image compression method.

Introduction

In the early 2000s, the image coding community was excited about Object-Based Image Coding (OBIC). The premise was simple: instead of compressing an entire rectangular grid of pixels uniformly, why not identify the important objects (like a person or a car) and compress them more effectively than the "unimportant" background? Despite its promise, OBIC largely failed to gain traction. The primary bottleneck was the "arbitrary shape" problem. Traditional codecs are built for blocks and squares; representing and processing complex, irregular shapes required specialized, computationally expensive tools that weren't practical at the time. Consequently, modern standards like HEVC-based BPG still treat images as a single layer, often resulting in muddy textures and lost details when forced into ultra-low bitrate scenarios (e.g., less than 0.1 bits per pixel).

Methodology: The LearntOBIC Framework

The authors introduce LearntOBIC, a framework that leverages the "masking" power of Convolutional Neural Networks (CNNs) to handle arbitrary shapes without needing dedicated shape-adaptive transforms.

The LearntOBIC architecture integrates segmentation with parallel compression branches for foreground and background layers.

Feature-Domain Masking

A key innovation of LearntOBIC is the use of feature-domain masking. The system uses a segmentation network (DeepLab with a ResNet-34 backbone) to generate a binary mask ($M$) that separates the foreground object from the background. Instead of just masking pixels, the authors apply the mask to the "latent features" within the compression network. By performing element-wise multiplication ($F \otimes M$), they can "activate" or "deactivate" specific feature elements. This allows the network to focus its "bit budget" precisely on the object's shape, avoiding the blurry boundaries often caused by pixel-domain masking where subsequent convolutions spread "zero" values from the background into the object.

Parallel Compression and Optimization

The masked foreground and background are processed by parallel branches of a Variational Autoencoder (VAE) based on the NLAIC (Non-Local Attention Image Compression) architecture. The entire pipeline—segmentation and compression—is optimized together using a Rate-Distortion Optimization (RDO) strategy. They use MS-SSIM (Multi-Scale Structural Similarity) as the primary distortion metric because it correlates better with human perception than standard PSNR at low bitrates.

Experimental Results

The model was trained and evaluated using the PASCAL VOC 2007 & 2012 datasets, with the Kodak dataset used to test generalization. At ultra-low bitrates (averaging ~0.065 bpp), LearntOBIC demonstrated clear advantages:

  • Quantitative Performance: On the PASCAL VOC test set, LearntOBIC achieved an MS-SSIM of 0.8992, outperforming JPEG2000 (0.8339) and BPG (0.8669).
  • Qualitative Quality: Visual comparisons show that while BPG and JPEG2000 produce heavy blocking artifacts or significant blurring on faces and textures, LearntOBIC preserves recognizable features and sharper edges. Even when the background is heavily degraded to save space, the primary object remains visually pleasant.
A qualitative reconstruction result using LearntOBIC at ultra-low bitrate, demonstrating preserved detail on the primary subject.

Conclusion

LearntOBIC successfully revisits object-based coding by integrating modern deep learning for segmentation and compression. By treating an image as distinct layers and optimizing them in an end-to-end neural framework, the proposed method achieves significantly better visual quality at ultra-low bitrates compared to traditional standards. Future work will address current limitations, such as the binary layer restriction and boundary artifacts, by exploring multi-layer decomposition and automated bit-allocation strategies. All relevant materials are publicly accessible at https://njuvision.github.io/Neural-Object-Coding/.

Summary updated

Abstract

The Object-Based Image Coding (OBIC) that was extensively studied about two decades ago, promised a vast application perspective for both ultra-low bitrate communication and high-level semantical content understanding, but it had rarely been used due to the inefficient compact representation of object with arbitrary shape. A fundamental issue behind is how to efficiently process the arbitrary-shaped objects at a fine granularity (e.g., feature element or pixel wise). To attack this, we have proposed to apply the element-wise masking and compression by devising an object segmentation network for image layer decomposition, and parallel convolution-based neural image compression networks to process masked foreground objects and background scene separately. All components are optimized in an end-to-end learning framework to intelligently weigh their (e.g., object and background) contributions for visually pleasant reconstruction. We have conducted comprehensive experiments to evaluate the performance on PASCAL VOC dataset at a very low bitrate scenario (e.g., $\lesssim$0.1 bits per pixel - bpp) which have demonstrated noticeable subjective quality improvement compared with JPEG2K, HEVC-based BPG and another learned image compression method. All relevant materials are made publicly accessible at https://njuvision.github.io/Neural-Object-Coding/.
BibTeX
@inproceedings{xia2020object,
  author = {Qi Xia and Haojie Liu and Zhan Ma},
  title = {{Object-Based Image Coding: A Learning-Driven Revisit}},
  booktitle = {ICME},
  year = {2020},
  url = {https://openreview.net/forum?id=90OKnRsFXF},
}
Figure