Opening the Black Box of Learned Image Coders

Zhihao Duan, Ming Lu, Zhan Ma, Fengqing Zhu

Abstract

End-to-end learned lossy image coders (LICs), as opposed to hand-crafted image codecs, have shown increasing superiority in terms of the rate-distortion performance. However, they are mainly treated as black-box systems and their interpretability is not well studied. In this paper, we show that LICs learn a set of basis functions to transform input image for its compact representation in the latent space, as analogous to the orthogonal transforms used in image coding standards. Our analysis provides insights to help understand how learned image coders work and could benefit future design and development.

Introduction

Learned Image Coders (LICs) are often dismissed as "black boxes," but recent research reveals they aren't as mysterious as they seem. By visualizing the internal representations of neural networks, it is possible to demonstrate that LICs actually learn non-linear "basis functions" that closely mirror the classical signal processing techniques—such as the Discrete Cosine Transform (DCT) or wavelets—used in traditional codecs like JPEG and HEVC.

In the world of image compression, two paradigms coexist: "hand-crafted" traditional codecs (JPEG, HEVC, VVC) and "learned" neural codecs. Traditional codecs are highly interpretable; they use linear orthogonal transforms to turn pixels into frequency coefficients, where every bit has a known representation. LICs, powered by deep convolutional neural networks, now outperform traditional codecs in rate-distortion performance. However, because they rely on complex, non-linear layers, they have largely been treated as black boxes. This work bridges that gap by asking whether these neural networks discover entirely new ways to compress data or simply implement a high-tech version of established principles.

Comparison between traditional linear orthogonal transforms and learned non-linear transformations in LICs, including examples of learned basis functions.

Methodology

To peek inside the decoder (synthesis transform) of various LICs, a heuristic solution involving three main techniques is proposed:

  1. Spatial and Channel-wise Decomposition: The compressed latent representation ($z$) is broken down into individual pieces—either by isolating a single spatial point or a single channel—and passed through the decoder. This allows for the visualization of the specific "pattern" each part of the compressed data is responsible for reconstructing.
  2. Defining "Artificial" Basis Functions: To remove dependency on specific input images, "channel impulses" ($\delta_i$) are created. By setting only one value in the latent space to a non-zero number and decoding it, the "impulse response" of the network is visualized. This output is defined as the basis function of the LIC.
  3. The Separability Hypothesis: The researchers tested whether the non-linear decoder behaves "linearly enough" to be interpreted this way by measuring the Mean Squared Error (MSE) between a standard reconstruction and a reconstruction formed by summing up individual decoded basis functions.
Illustration and results of spatial and channel-wise decomposition methods used to interpret LIC latent representations.

Key Insights

Visualizing these basis functions reveals several critical properties of LICs:

  • Neural Networks as Signal Processors: The basis functions learned by LICs look remarkably like classical transform bases, such as Walsh-Hadamard transforms, Haar wavelets, and edge detectors.
  • Automatic Luma/Chroma Separation: Without explicit instruction, LICs trained on color images automatically learn to separate brightness (luma) from color (chroma) information, mimicking the RGB-to-YCbCr conversion used in JPEG.
  • Architecture Invariance: These basis patterns appear consistently across different network architectures, bit rates, and loss functions (MSE vs. MS-SSIM), suggesting they represent an "optimal" way to represent image data.
  • The Non-Linear KLT: LICs essentially function as a non-linear counterpart to the Karhunen-Loève Transform (KLT), the theoretically optimal linear transform for a given data distribution.

Results and Validation

The "separability hypothesis" was validated with extremely low MSE (less than 0.005 in all cases), proving that despite the networks' non-linearity, latent coefficients are largely independent and their contributions to the final image are additive. Visual evidence shows "checkerboard" patterns for high frequencies, smooth gradients for low frequencies, and specific color-blob patterns for chroma channels. More recent, high-performing models show more sophisticated basis functions that resemble 2-D Walsh-Hadamard transforms, explaining their superior compression performance.

Conclusion

While this work provides strong empirical evidence, a rigorous mathematical proof linking LIC transformations to the Karhunen-Loève Transform (KLT) is still required. Future work should also examine the encoder's role in selecting these bases. These insights aim to move the field toward designing network architectures that explicitly leverage discovered basis-decomposition properties rather than relying solely on blind data-driven training.

Summary updated

BibTeX
@inproceedings{duan2022opening,
  author = {Zhihao Duan and Ming Lu and Zhan Ma and Fengqing Zhu},
  title = {{Opening the Black Box of Learned Image Coders}},
  booktitle = {PCS},
  year = {2022},
  doi = {10.1109/pcs56426.2022.10018031},
}
Figure