TL;DR
Researchers have revived the decades-old concept of Object-Based Image Coding (OBIC) by integrating modern deep learning for segmentation and compression. By treating an image as distinct layers (foreground objects vs. background) and optimizing them in an end-to-end neural framework, the proposed LearntOBIC achieves significantly better visual quality at ultra-low bitrates compared to traditional standards like JPEG2000 and BPG.
What Problem Does This Paper Address?
In the early 2000s, the image coding community was excited about Object-Based Image Coding (OBIC). The idea was simple: instead of compressing an entire rectangular grid of pixels uniformly, why not identify the important objects (like a person or a car) and compress them more effectively than the "unimportant" background?
Despite its promise, OBIC largely failed to gain traction. The primary bottleneck was the "arbitrary shape" problem. Traditional codecs are built for blocks and squares; representing and processing complex, irregular shapes required specialized, computationally expensive tools that weren't practical at the time. Consequently, modern standards like HEVC-based BPG still treat images as a single layer, often resulting in muddy textures and lost details when forced into ultra-low bitrate scenarios (e.g., less than 0.1 bits per pixel).
Methods
The authors introduce LearntOBIC, a framework that leverages the "masking" power of Convolutional Neural Networks (CNNs) to handle arbitrary shapes without needing dedicated shape-adaptive transforms.
- Layer Decomposition: The system uses a segmentation network (DeepLab with a ResNet-34 backbone) to generate a binary mask ($M$) that separates the foreground object from the background.
- Feature-Domain Masking: Instead of just masking pixels, the authors apply the mask to the "latent features" within the compression network. By performing element-wise multiplication ($F \otimes M$), they can "activate" or "deactivate" specific feature elements. This allows the network to focus its "bit budget" precisely on the object's shape.
- Parallel Compression: The masked foreground and background are processed by parallel branches of a Variational Autoencoder (VAE) based on the NLAIC (Non-Local Attention Image Compression) architecture.
- End-to-End Optimization: The entire pipeline—segmentation and compression—is optimized together using a Rate-Distortion Optimization (RDO) strategy. They use MS-SSIM (Multi-Scale Structural Similarity) as the primary distortion metric because it correlates better with human perception than standard PSNR at low bitrates.
Datasets
The model was trained and evaluated using:
- PASCAL VOC 2007 & 2012: Used for training the segmentation and compression components.
- Kodak Dataset: Used to test the model’s generalization ability on images outside the training distribution.
Key Insights and Ideas
- The "Aha" Moment for Masking: The authors found that masking in the feature domain is superior to masking in the pixel domain. Pixel-domain masking often results in blurry object boundaries because subsequent convolutions spread the "zero" values from the background into the object. Masking the latent features avoids this "bleeding" effect.
- Intelligent Bit Allocation: By adjusting weighting parameters ($a_1$ and $a_2$) in the loss function, the system can intentionally "shift" bits from the background to the foreground. This ensures that even at extreme compression ratios, the most semantically important parts of the image remain sharp.
- Efficiency over Accuracy: Interestingly, the authors found that perfect segmentation isn't strictly necessary for good compression. Switching from a heavy ResNet-101 backbone to a lighter ResNet-34 for segmentation halved the computational cost with negligible impact on compression efficiency.
Key Results
At ultra-low bitrates (averaging ~0.065 bpp), LearntOBIC demonstrated clear advantages:
- Quantitative: On the PASCAL VOC test set, LearntOBIC achieved an MS-SSIM of 0.8992, outperforming JPEG2000 (0.8339) and BPG (0.8669).
- Qualitative: Visual comparisons show that while BPG and JPEG2000 produce heavy blocking artifacts or significant blurring on faces and textures, LearntOBIC preserves recognizable features and sharper edges. Even when the background is heavily degraded to save space, the primary object remains visually pleasant.
Limitations and Future Work
While a successful "revisit," the paper notes several areas for improvement:
- Binary Limitation: The current implementation only supports two layers (one foreground, one background). Real-world scenes often have multiple overlapping objects that would benefit from a multi-layer approach.
- Boundary Artifacts: While feature-domain masking is better, some artifacts still occur at the edges of objects due to the nature of convolutional padding and receptive fields.
- Automation: Currently, the bit-shifting weights are set manually. Future work could involve an automated system that decides how to distribute bits based on the specific task (e.g., human viewing vs. machine vision).
- Segmentation Robustness: The system's performance is tethered to the quality of the segmentation mask; if the segmentation fails, the compression strategy may prioritize the wrong areas.
Summary updated