Transformer-Based Variable-Rate Image Compression with Region-of-Interest Control

Chia-Hao Kao, Ying-Chieh Weng, Yi-Hsin Chen, Wei-Chen Chiu, Wen-Hsiao Peng

TL;DR

Researchers have developed a new transformer-based image compression system that uses "prompt tuning" to achieve both variable-rate compression and region-of-interest (ROI) control within a single model. By introducing content-adaptive prompt tokens, the system can precisely balance bit-rate and image quality across different areas of an image with lower computational complexity than previous state-of-the-art methods.

What Problem Does This Paper Address?

In the world of learned image compression, two features are highly desirable but difficult to combine efficiently: variable-rate coding (adjusting the file size/quality trade-off on the fly) and Region-of-Interest (ROI) control (allocating more bits to important parts of an image, like a face or a foreground object).

While Convolutional Neural Networks (CNNs) have established methods for this, Transformers—which are currently setting new benchmarks in compression performance—lack a flexible way to handle both. Existing approaches often rely on "scaling" feature maps or using "Spatial Feature Transforms" (SFT). However, these methods can be computationally expensive or struggle to "disentangle" rate control from spatial quality control, making it hard to specify, for example, a high-quality foreground at a low overall bit-rate.

Methods

The authors introduce a Prompted Swin-transformer Block (P-STB). Inspired by "prompt tuning" in Natural Language Processing, they don't just feed the image into the transformer; they also feed in "prompts"—special tokens that act as instructions for the model.

Key components include:

  • Prompt Generation Networks ($p_a, p_s$): These small sub-networks take the input image, a "lambda map" (representing the desired bit-rate), and an ROI mask. They generate content-adaptive prompt tokens.
  • Disentangled Control: Because the rate parameter ($\lambda$) and the ROI mask ($M_R$) are separate inputs to the prompt generator, the model learns to treat them as independent dimensions. You can change the overall bit-rate without losing the relative importance of your ROI.
  • Augmented Attention: In the transformer's self-attention mechanism, these prompt tokens are concatenated to the Key ($K$) and Value ($V$) matrices. This allows the image tokens to "attend" to the prompts, effectively receiving instructions on how much detail to preserve in specific spatial locations.
  • Efficiency: To keep complexity low, the prompt tokens are generated at a lower resolution (one-fourth the number of image tokens), reducing the overhead of the attention mechanism.

Datasets

The model was trained and evaluated using several standard benchmarks:

  • Training: Flicker2W and COCO 2017 training sets.
  • Evaluation (General): The Kodak dataset was used to test standard variable-rate performance without ROI.
  • Evaluation (ROI): The COCO 2017 validation set, using ground-truth segmentation masks to define ROIs.

Key Insights and Ideas

  • Prompts as Adaptive Controllers: Instead of modifying the weights of the main compression network (which is slow and difficult), the authors use prompts to "steer" the existing weights. This makes the model highly flexible without needing to retrain for different bit-rates.
  • Disentanglement is Key: By separating the rate parameter from the ROI mask, the user gets an intuitive "knob" for quality and a "map" for importance. Previous methods often lumped these into a single "quality map," which made fine-grained control difficult.
  • Spatial Awareness: Visualization of the attention maps shows that prompt tokens naturally learn to focus on ROI boundaries, proving that the model is successfully using the prompts to guide bit allocation spatially.

Key Results

  • Performance: The proposed method (SWIN+PT) performs comparably to training multiple separate models for different bit-rates (the "gold standard" for quality, but impractical for storage).
  • Superiority over SFT: It outperforms the common Spatial Feature Transform (SFT) approach in both rate-distortion performance and subjective visual quality, especially in blurring backgrounds to save bits for the foreground.
  • Computational Efficiency: Despite having more parameters (32.7M vs 27.5M for CNN+SFT), the model is more efficient in terms of operations. It requires 1070.68 kMACs/pixel, significantly lower than the 1480.16 required by the CNN+SFT baseline. This is because the conditioning signal is processed at a lower resolution.

Limitations and Future Work

  • Model Size: While computationally efficient in terms of FLOPs/MACs, the model has a higher total parameter count than some baselines. This might be a concern for devices with very limited memory.
  • Entropy Coding: The current architecture (built on the TIC framework) does not include a context model for entropy coding to keep the focus on the prompt mechanism. Integrating a more advanced entropy model could further improve compression gains.
  • Task-Specific ROI: The paper focuses on user-defined or segmentation-based ROIs. Future work could explore automatically generating ROI masks based on downstream tasks, like object detection or facial recognition.

Summary updated

Abstract

This paper proposes a transformer-based learned image compression system. It is capable of achieving variable-rate compression with a single model while supporting the region-of-interest (ROI) functionality. Inspired by prompt tuning, we introduce prompt generation networks to condition the transformer-based autoencoder of compression. Our prompt generation networks generate content-adaptive tokens according to the input image, an ROI mask, and a rate parameter. The separation of the ROI mask and the rate parameter allows an intuitive way to achieve variable-rate and ROI coding simultaneously. Extensive experiments validate the effectiveness of our proposed method and confirm its superiority over the other competing methods.
BibTeX
@inproceedings{kao2023transformer,
  author = {Chia-Hao Kao and Ying-Chieh Weng and Yi-Hsin Chen and Wei-Chen Chiu and Wen-Hsiao Peng},
  title = {{Transformer-Based Variable-Rate Image Compression with Region-of-Interest Control}},
  booktitle = {ICIP},
  year = {2023},
  url = {https://openreview.net/forum?id=PsJlCdbJLq},
}

Loading related content...

Figure