Visual Analysis Motivated Rate-Distortion Model for Image Coding

Zhimeng Huang, Chuanmin Jia, Shanshe Wang, Siwei Ma

Abstract

Standard image compression standards, such as Versatile Video Coding (VVC), are primarily optimized for human perceptual quality. However, as machine vision tasks like autonomous driving and surveillance become more prevalent, traditional metrics like Mean Square Error (MSE) often fail to preserve the semantic features necessary for AI models. This paper introduces a machine-vision-optimized rate-distortion model that prioritizes bit allocation for regions critical to machine analysis. By incorporating a Region of Interest for Machine (ROIM) map and a Multi-Scale Feature Distortion (MSFD) metric into the VVC framework, the proposed method achieves significant bitrate reductions—up to 28.17%—while maintaining high performance on tasks such as object detection and semantic segmentation.

Introduction

Most existing image codecs are designed to minimize pixel-level distortion or maximize human-perceived quality. While effective for human consumption, these methods can be detrimental to Convolutional Neural Networks (CNNs). At low bitrates, standard compression artifacts can lead to a significant drop in accuracy; for instance, VGG-19 accuracy on ImageNet can fall significantly when bitrates are low (e.g., ~68% Top-5 accuracy at 0.1 bpp). Machines prioritize semantic features over visual smoothness. This research addresses the gap by optimizing the Versatile Video Coding (VVC) standard for machine vision tasks.

Proposed Methodology

The proposed framework integrates three core innovations into the VTM encoder to bridge the gap between compression and machine analysis.

The proposed VTM Encoder framework featuring ROIM generation, bit allocation strategy, and the MSFD distortion measurement model.

1. Region of Interest for Machine (ROIM)

The system utilizes a pre-trained Region Proposal Network (RPN) to generate importance maps. By calculating the density of bounding boxes for each Coding Tree Unit (CTU), the encoder identifies regions most critical for downstream machine tasks.

2. Visual Analysis-Friendly Bit Allocation

Using ROIM data, the encoder intelligently distributes the bit budget. Critical regions, such as vehicles or pedestrians, receive higher bitrates, while background elements like the sky are compressed more aggressively. A "connectivity" constraint is also applied to maintain consistent Quantization Parameters (QP) across object boundaries, preventing artifacts that might confuse AI models.

3. Multi-Scale Feature Distortion (MSFD)

Traditional Rate-Distortion Optimization (RDO) relies on MSE. This paper introduces MSFD, a deep-feature-based metric. Since standard VVC coding units (as small as 4x4) lack sufficient spatial context for CNNs, the MSFD approach uses a multi-scale window to incorporate surrounding reconstructed pixels. This expands the receptive field, allowing the model to calculate distortion based on the preservation of high-level features extracted via a VGG-11 network.

Experimental Results

The model was evaluated across three tasks: image classification (ImageNet), object detection (VOC-2007), and semantic segmentation (COCO-2014).

Top-1 and Top-5 accuracy curves relative to bitrate (bpp), demonstrating the performance of the proposed model.

The proposed scheme significantly outperforms the VVC test model (VTM 10.1) in terms of BD-Rate savings for equivalent accuracy levels:

  • Semantic Segmentation: 28.17% reduction.
  • Object Detection: 26.25% reduction.
  • Image Classification: ~19% reduction.

Qualitative analysis shows that the model successfully preserves identifying features that standard VVC might blur, preventing misclassifications (e.g., mistaking an iPod for a television) at low bitrates.

Conclusion and Future Work

This research demonstrates that optimizing image compression for semantic feature preservation rather than pixel fidelity can yield substantial efficiency gains for machine vision. While currently limited to VVC intra-frame coding, future work will explore extending this model to inter-frame video coding. Additionally, reducing the computational overhead of the CNN-based RDO loop remains a key area for optimization.

Summary updated

BibTeX
@inproceedings{huang2021visual,
  author = {Zhimeng Huang and Chuanmin Jia and Shanshe Wang and Siwei Ma},
  title = {{Visual Analysis Motivated Rate-Distortion Model for Image Coding}},
  booktitle = {ICME},
  year = {2021},
  doi = {10.1109/icme51207.2021.9428417},
}
Figure