Object-Based Image Coding: A Learning-Driven Revisit

Qi Xia, Haojie Liu, Zhan Ma

TL;DR

Researchers have revived the decades-old concept of Object-Based Image Coding (OBIC) by integrating modern deep learning for segmentation and compression. By treating an image as distinct layers (foreground objects vs. background) and optimizing them in an end-to-end neural framework, the proposed LearntOBIC achieves significantly better visual quality at ultra-low bitrates compared to traditional standards like JPEG2000 and BPG.

What Problem Does This Paper Address?

In the early 2000s, the image coding community was excited about Object-Based Image Coding (OBIC). The idea was simple: instead of compressing an entire rectangular grid of pixels uniformly, why not identify the important objects (like a person or a car) and compress them more effectively than the "unimportant" background?

Despite its promise, OBIC largely failed to gain traction. The primary bottleneck was the "arbitrary shape" problem. Traditional codecs are built for blocks and squares; representing and processing complex, irregular shapes required specialized, computationally expensive tools that weren't practical at the time. Consequently, modern standards like HEVC-based BPG still treat images as a single layer, often resulting in muddy textures and lost details when forced into ultra-low bitrate scenarios (e.g., less than 0.1 bits per pixel).

Methods

The authors introduce LearntOBIC, a framework that leverages the "masking" power of Convolutional Neural Networks (CNNs) to handle arbitrary shapes without needing dedicated shape-adaptive transforms.

  1. Layer Decomposition: The system uses a segmentation network (DeepLab with a ResNet-34 backbone) to generate a binary mask ($M$) that separates the foreground object from the background.
  2. Feature-Domain Masking: Instead of just masking pixels, the authors apply the mask to the "latent features" within the compression network. By performing element-wise multiplication ($F \otimes M$), they can "activate" or "deactivate" specific feature elements. This allows the network to focus its "bit budget" precisely on the object's shape.
  3. Parallel Compression: The masked foreground and background are processed by parallel branches of a Variational Autoencoder (VAE) based on the NLAIC (Non-Local Attention Image Compression) architecture.
  4. End-to-End Optimization: The entire pipeline—segmentation and compression—is optimized together using a Rate-Distortion Optimization (RDO) strategy. They use MS-SSIM (Multi-Scale Structural Similarity) as the primary distortion metric because it correlates better with human perception than standard PSNR at low bitrates.

Datasets

The model was trained and evaluated using:

  • PASCAL VOC 2007 & 2012: Used for training the segmentation and compression components.
  • Kodak Dataset: Used to test the model’s generalization ability on images outside the training distribution.

Key Insights and Ideas

  • The "Aha" Moment for Masking: The authors found that masking in the feature domain is superior to masking in the pixel domain. Pixel-domain masking often results in blurry object boundaries because subsequent convolutions spread the "zero" values from the background into the object. Masking the latent features avoids this "bleeding" effect.
  • Intelligent Bit Allocation: By adjusting weighting parameters ($a_1$ and $a_2$) in the loss function, the system can intentionally "shift" bits from the background to the foreground. This ensures that even at extreme compression ratios, the most semantically important parts of the image remain sharp.
  • Efficiency over Accuracy: Interestingly, the authors found that perfect segmentation isn't strictly necessary for good compression. Switching from a heavy ResNet-101 backbone to a lighter ResNet-34 for segmentation halved the computational cost with negligible impact on compression efficiency.

Key Results

At ultra-low bitrates (averaging ~0.065 bpp), LearntOBIC demonstrated clear advantages:

  • Quantitative: On the PASCAL VOC test set, LearntOBIC achieved an MS-SSIM of 0.8992, outperforming JPEG2000 (0.8339) and BPG (0.8669).
  • Qualitative: Visual comparisons show that while BPG and JPEG2000 produce heavy blocking artifacts or significant blurring on faces and textures, LearntOBIC preserves recognizable features and sharper edges. Even when the background is heavily degraded to save space, the primary object remains visually pleasant.

Limitations and Future Work

While a successful "revisit," the paper notes several areas for improvement:

  • Binary Limitation: The current implementation only supports two layers (one foreground, one background). Real-world scenes often have multiple overlapping objects that would benefit from a multi-layer approach.
  • Boundary Artifacts: While feature-domain masking is better, some artifacts still occur at the edges of objects due to the nature of convolutional padding and receptive fields.
  • Automation: Currently, the bit-shifting weights are set manually. Future work could involve an automated system that decides how to distribute bits based on the specific task (e.g., human viewing vs. machine vision).
  • Segmentation Robustness: The system's performance is tethered to the quality of the segmentation mask; if the segmentation fails, the compression strategy may prioritize the wrong areas.

Summary updated

Abstract

The Object-Based Image Coding (OBIC) that was extensively studied about two decades ago, promised a vast application perspective for both ultra-low bitrate communication and high-level semantical content understanding, but it had rarely been used due to the inefficient compact representation of object with arbitrary shape. A fundamental issue behind is how to efficiently process the arbitrary-shaped objects at a fine granularity (e.g., feature element or pixel wise). To attack this, we have proposed to apply the element-wise masking and compression by devising an object segmentation network for image layer decomposition, and parallel convolution-based neural image compression networks to process masked foreground objects and background scene separately. All components are optimized in an end-to-end learning framework to intelligently weigh their (e.g., object and background) contributions for visually pleasant reconstruction. We have conducted comprehensive experiments to evaluate the performance on PASCAL VOC dataset at a very low bitrate scenario (e.g., ≲ 0.1 bits per pixel - bpp) which have demonstrated noticeable subjective quality improvement compared with JPEG2K, HEVC-based BPG and another learned image compression method. All relevant materials are made publicly accessible at https://njuvision.github.io/ Neural-Object-Coding/.
BibTeX
@inproceedings{xia2020object,
  author = {Qi Xia and Haojie Liu and Zhan Ma},
  title = {{Object-Based Image Coding: A Learning-Driven Revisit}},
  booktitle = {ICME},
  year = {2020},
  url = {https://openreview.net/forum?id=90OKnRsFXF},
}

Loading related content...

Figure