Contextformer: A Transformer with Spatio-Channel Attention for Context Modeling in Learned Image Compression

Ahmet Burakhan Koyuncu, Han Gao, Atanas Boev, Georgii Gaikov, Elena Alshina, Eckehard Steinbach

Abstract

Entropy modeling is a key component for high-performance image compression algorithms. Recent developments in autoregressive context modeling helped learning-based methods to surpass their classical counterparts. However, the performance of those models can be further improved due to the underexploited spatio-channel dependencies in latent space, and the suboptimal implementation of context adaptivity. Inspired by the adaptive characteristics of the transformers, we propose a transformer-based context model, named Contextformer, which generalizes the de facto standard attention mechanism to spatio-channel attention. We replace the context model of a modern compression framework with the Contextformer and test it on the widely used Kodak, CLIC2020, and Tecnick image datasets. Our experimental results show that the proposed model provides up to 11% rate savings compared to the standard Versatile Video Coding (VVC) Test Model (VTM) 16.2, and outperforms various learning-based models in terms of PSNR and MS-SSIM.

Addressing the Adaptivity Gap in Entropy Modeling

In learned image compression, entropy modeling estimates the probability distribution of the latent space representation to enable efficient storage. Most modern systems rely on autoregressive context models that predict a pixel's value based on previously decoded neighbors. However, existing models face two major hurdles:

  1. Lack of Adaptivity: Traditional models often use masked Convolutional Neural Networks (CNNs). Because CNNs are location-agnostic, they apply fixed kernels regardless of image content, failing to adapt to local textures or patterns.
  2. Underutilized Dependencies: Many models focus primarily on spatial neighbors, neglecting the rich dependencies across feature channels. Since channels in a latent representation often carry over 60% of the total information, failing to model these relationships leads to significant coding inefficiency.

The Contextformer Architecture

To overcome these limitations, the authors introduce Contextformer, a transformer architecture designed to serve as the context model within an end-to-end compression framework. Unlike standard Vision Transformers that treat spatial patches as tokens, Contextformer treats "spatio-channel patches" (1x1 spatial pixels across a segment of channels) as tokens. This allows the model to attend to both distant spatial locations and related feature channels simultaneously.

The Contextformer architecture (a) integrated into a compression framework, alongside sequence generation strategies for single (b) and multiple (c) channel segments.

Spatio-Channel Attention and Causal Masking

A critical requirement for autoregressive decoding is causality: the model cannot look at data that hasn't been decoded yet. Contextformer implements a 3D causal mask to ensure the "query" only attends to previously processed "keys" and "values."

Visualization of various 3D spatio-channel attention masking patterns used to maintain causality during autoregressive decoding.

The researchers explored two primary coding orders:

  • Spatial-First-Order (SFO): Decodes a single channel segment across all spatial locations before moving to the next segment.
  • Channel-First-Order (CFO): Prioritizes decoding all channel segments at one spatial location before moving to the next.

Experimentation revealed that CFO is more effective, suggesting that in latent space, the relationship between different features at the same location is often more predictive than the relationship between neighboring pixels.

Performance and Efficiency

The model was trained on the Vimeo-90K dataset and evaluated on standard benchmarks including Kodak, CLIC2020, and Tecnick.

Key Results:

  • Superior Compression: Contextformer achieved 6.9% to 10.5% bit-rate savings over VTM 16.2 (the VVC reference software) across all datasets, reaching up to 11% savings on the Tecnick dataset.
  • State-of-the-Art Fidelity: It consistently outperformed leading learned models in both PSNR (signal fidelity) and MS-SSIM (perceptual quality).
  • Runtime Optimizations: While transformers are computationally intensive, the authors introduced several optimizations, including Wavefront Coding, which enabled 9x faster decoding compared to traditional 3D context models by processing independent blocks along a diagonal.

Conclusion and Future Work

Contextformer demonstrates that the dynamic, content-aware nature of transformer attention is highly effective for modeling the complex dependencies in compressed latent spaces. While the quadratic complexity of attention remains a challenge for high-resolution (4K) images, the significant gains in compression efficiency mark a major step forward for learned image codecs. Future research will focus on more efficient attention mechanisms, such as linear or sparse attention, to further reduce the computational overhead for mobile and real-time applications.

Summary updated

Abstract

Entropy modeling is a key component for high-performance image compression algorithms. Recent developments in autoregressive context modeling helped learning-based methods to surpass their classi- cal counterparts. However, the performance of those models can be fur- ther improved due to the underexploited spatio-channel dependencies in latent space, and the suboptimal implementation of context adaptivity. Inspired by the adaptive characteristics of the transformers, we propose a transformer-based context model, named Contextformer, which general- izes the de facto standard attention mechanism to spatio-channel atten- tion. We replace the context model of a modern compression framework with the Contextformer and test it on the widely used Kodak, CLIC2020, and Tecnick image datasets. Our experimental results show that the pro- posed model provides up to 11% rate savings compared to the standard Versatile Video Coding (VVC) Test Model (VTM) 16.2, and outperforms various learning-based models in terms of PSNR and MS-SSIM.
BibTeX
@inproceedings{koyuncu2022contextformer,
  author = {Ahmet Burakhan Koyuncu and Han Gao and Atanas Boev and Georgii Gaikov and Elena Alshina and Eckehard Steinbach},
  title = {{Contextformer: A Transformer with Spatio-Channel Attention for Context Modeling in Learned Image Compression}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-19800-7\_26},
}
Figure