Abstract
This paper introduces a transformer-based learned image compression system designed to achieve variable-rate compression and region-of-interest (ROI) control within a single model. Inspired by prompt tuning, the architecture utilizes prompt generation networks to condition a transformer-based autoencoder. These networks generate content-adaptive tokens based on the input image, an ROI mask, and a rate parameter. By separating the ROI mask and the rate parameter, the system provides an intuitive mechanism for simultaneous variable-rate and ROI coding. Experimental results demonstrate that this method outperforms existing approaches in both rate-distortion performance and computational efficiency.
Introduction
In the field of learned image compression, two features are highly sought after: variable-rate coding and Region-of-Interest (ROI) control. Variable-rate coding allows a single model to adjust the trade-off between file size and image quality dynamically. ROI control enables the allocation of more bits to critical areas of an image—such as faces or foreground objects—while compressing less important regions more aggressively.
While Convolutional Neural Networks (CNNs) have established techniques for these features, Transformers—which currently define the state-of-the-art in compression—lack a flexible framework to handle both simultaneously. Existing Transformer-based methods often rely on scaling feature maps or Spatial Feature Transforms (SFT), which can be computationally expensive and often fail to "disentangle" rate control from spatial quality control. This makes it difficult for users to specify high-quality foregrounds at low overall bit-rates.
Proposed Method: Prompted Swin-Transformers
The core innovation of this work is the Prompted Swin-transformer Block (P-STB). Drawing inspiration from "prompt tuning" in Natural Language Processing, the researchers introduce "prompts"—special tokens that serve as instructions for the model—rather than simply feeding the image data through the transformer layers.
Prompt Generation Networks
The system employs two small sub-networks, $p_a$ and $p_s$, to generate content-adaptive prompt tokens. These networks take three inputs:
- The input image.
- A "lambda map" (representing the desired bit-rate).
- An ROI mask ($M_R$).
By keeping the rate parameter and the ROI mask as separate inputs, the model learns to treat them as independent dimensions. This "disentangled control" allows a user to adjust the global bit-rate without losing the relative importance assigned to the ROI.
Augmented Attention Mechanism
In a standard transformer, self-attention is calculated using Query (Q), Key (K), and Value (V) matrices. In the P-STB, the generated prompt tokens are concatenated to the K and Value matrices. This allows the image tokens to "attend" to the prompts, effectively receiving spatial instructions on how much detail to preserve. To maintain computational efficiency, these prompt tokens are generated at a lower resolution (one-fourth the number of image tokens), significantly reducing the overhead of the attention mechanism.
Experimental Setup
The model was trained on the Flicker2W and COCO 2017 datasets. For evaluation, the researchers used:
- Kodak Dataset: To test standard variable-rate performance without ROI.
- COCO 2017 Validation Set: To test ROI-specific performance using ground-truth segmentation masks.
Results and Discussion
The proposed method, referred to as SWIN+PT, demonstrates several key advantages over traditional baselines and CNN-based SFT methods.
Rate-Distortion Performance
SWIN+PT performs comparably to the "gold standard" of training multiple separate models for different bit-rates. This is a significant achievement, as it provides the flexibility of a variable-rate model without the storage overhead of multiple fixed-rate models. Furthermore, it outperforms the common SFT approach in both rate-distortion metrics and subjective visual quality, particularly in its ability to intelligently blur backgrounds to save bits for the foreground.
Computational Efficiency
Despite a higher parameter count (32.7M compared to 27.5M for CNN+SFT), the SWIN+PT model is more operationally efficient. It requires 1070.68 kMACs/pixel, which is significantly lower than the 1480.16 kMACs/pixel required by the CNN+SFT baseline. This efficiency stems from processing the conditioning signals (prompts) at a lower resolution.
Spatial Awareness
Visualization of the model's attention maps reveals that the prompt tokens naturally focus on ROI boundaries. This confirms that the model successfully uses the prompts to guide bit allocation spatially, validating the "steering" effect of the prompt-tuning mechanism.
Conclusion and Future Work
This research presents a robust framework for integrating variable-rate and ROI control into Transformer-based image compression. By using content-adaptive prompts, the model achieves high-quality results with lower computational complexity than previous state-of-the-art methods.
While the model is efficient in terms of FLOPs, its total parameter count remains a consideration for memory-constrained devices. Future improvements could include integrating advanced entropy models to further enhance compression gains or exploring automated ROI mask generation based on downstream tasks like object detection or facial recognition.
Summary updated