Compressing image encoders via latent distillation

Caroline Mazini Rodrigues, Nicolas Keriven, Wided Miled

TL;DR

This paper introduces Latent Distillation, a simplified strategy to shrink image encoders for resource-constrained edge devices without sacrificing reconstruction quality. By training a "student" encoder to mimic the latent representations of a "teacher" model while keeping the decoder frozen, the authors achieve up to a 35x reduction in computational complexity using as little as 0.1% of the original training data.

What Problem Does This Paper Address?

End-to-end Learned Image Coders (LICs) have surpassed traditional codecs like JPEG in performance, but they come with a heavy "tax": they are computationally expensive and require massive datasets to train.

While much research has focused on making decoders lightweight (for fast playback on phones), the encoder is often overlooked. In Internet-of-Things (IoT) and edge computing scenarios, the encoder lives on the hardware-limited device. Simply shrinking the encoder's layers (reducing "width") usually leads to a massive drop in performance because the model loses the capacity to learn complex features from scratch. The authors tackle the challenge of creating tiny, efficient encoders that can be trained quickly with very limited data.

Methods

The authors propose a Feature-Based Knowledge Distillation (KD) approach. Instead of training a small encoder from scratch using the standard, complex Rate-Distortion loss, they use a pre-trained "Teacher" model to guide a "Student" model.

  1. Architecture: They take a standard architecture (like the Factorized Prior or MS-ILLM) and create a student version by reducing the number of filters in the encoder's convolutional layers by factors of 2, 4, or 8.
  2. Frozen Decoder: Crucially, they keep the Teacher’s high-quality decoder frozen. The student encoder must learn to produce latent representations that the existing decoder already understands.
  3. Latent Distillation Loss: To simplify training, they discard the usual reconstruction loss ($L_2$ or MS-SSIM). Instead, the student is trained solely to minimize the Mean Squared Error (MSE) between its output latents ($y_S$) and the teacher’s latents ($y_T$). For hyperprior models, they also match the "hyper-latents" ($z_S$ and $z_T$).
  4. Resource Efficiency: This "latent matching" is much easier to optimize than the original task, allowing the model to converge faster and with significantly less data.

Datasets

  • Training: Subsets of the Vimeo-90k dataset (originally ~89k video clips). The authors tested varying proportions ($\rho$), ranging from 10% down to a tiny 0.1% of the data.
  • Evaluation: The classic Kodak dataset (24 images) and the CLIC2020 test split (428 images).
  • Teacher Models: Pre-trained on Vimeo-90k (for Factorized Prior) and OpenImages V6 (for the generative MS-ILLM model).

Key Insights and Ideas

  • The "Aha" Moment on Data: The most striking finding is how little data is needed. A student encoder trained via KD on just 0.1% of the dataset significantly outperforms a student trained from scratch on the same small slice. The teacher’s latents act as a "dense" supervisory signal that is much more informative than raw pixels.
  • Stability through Guidance: Training reduced-width models from scratch is often unstable. Latent distillation provides a stable target, leading to faster convergence and better "statistical fidelity" (the images look more natural and less "blurry").
  • Asymmetric Compression: This method enables an asymmetric ecosystem where a very cheap, low-power encoder at the edge can communicate perfectly with a heavy, high-performance decoder in the cloud.
  • Simplified Loss works: You don't actually need to calculate the bit-rate or reconstruction error during distillation training; simply mimicking the teacher's bottleneck features is a sufficient proxy for the entire compression task.

Key Results

  • Massive Speedups: For the Factorized Prior model, a $\div 8$ reduction in encoder width resulted in a 35x reduction in MACs (Multiply-Accumulate Operations) and a 16x reduction in storage size (from 6.0 MB down to 0.37 MB).
  • Superior Quality: Across all tests, the KD-trained students achieved higher PSNR (signal-to-noise ratio) and lower FID (a measure of visual quality/naturalness) than models trained from scratch with the same reduced width.
  • Bit-rate Savings: Using Bjøntegaard Delta (BD-Rate) metrics, the KD approach showed significant bit-rate reductions (up to 77% improvement in some cases) compared to the "frozen" baseline trained without distillation.

Limitations and Future Work

  • Shared Skeleton: The current method assumes the student and teacher share a similar architectural "skeleton" (just with different widths). It’s unclear how well this would work if the student had a completely different internal structure.
  • Encoder-Only: The paper focuses strictly on the encoder. While this is great for IoT upload scenarios, the decoder remains large.
  • Future Directions: The authors explicitly mention that their next step is to explore decoder reduction, potentially creating a fully distilled, end-to-end lightweight pipeline. They also aim to test the method on even more complex generative compression models.

Summary updated

Abstract

Deep learning models for image compression often face practical limitations in hardware-constrained applications. Although these models achieve high-quality reconstructions, they are typically complex, heavyweight, and require substantial training data and computational resources. We propose a methodology to partially compress these networks by reducing the size of their encoders. Our approach uses a simplified knowledge distillation strategy to approximate the latent space of the original models with less data and shorter training, yielding lightweight encoders from heavyweight ones. We evaluate the resulting lightweight encoders across two different architectures on the image compression task. Experiments show that our method preserves reconstruction quality and statistical fidelity better than training lightweight encoders with the original loss, making it practical for resource-limited environments.
BibTeX
@inproceedings{rodrigues2026compressing,
  author = {Caroline Mazini Rodrigues and Nicolas Keriven and Wided Miled},
  title = {{Compressing image encoders via latent distillation}},
  booktitle = {SPIRE},
  year = {2026},
}
Figure