TL;DR
This paper introduces Latent Distillation, a simplified strategy to shrink image encoders for resource-constrained edge devices without sacrificing reconstruction quality. By training a "student" encoder to mimic the latent representations of a "teacher" model while keeping the decoder frozen, the authors achieve up to a 35x reduction in computational complexity using as little as 0.1% of the original training data.
What Problem Does This Paper Address?
End-to-end Learned Image Coders (LICs) have surpassed traditional codecs like JPEG in performance, but they come with a heavy "tax": they are computationally expensive and require massive datasets to train.
While much research has focused on making decoders lightweight (for fast playback on phones), the encoder is often overlooked. In Internet-of-Things (IoT) and edge computing scenarios, the encoder lives on the hardware-limited device. Simply shrinking the encoder's layers (reducing "width") usually leads to a massive drop in performance because the model loses the capacity to learn complex features from scratch. The authors tackle the challenge of creating tiny, efficient encoders that can be trained quickly with very limited data.
Methods
The authors propose a Feature-Based Knowledge Distillation (KD) approach. Instead of training a small encoder from scratch using the standard, complex Rate-Distortion loss, they use a pre-trained "Teacher" model to guide a "Student" model.
- Architecture: They take a standard architecture (like the Factorized Prior or MS-ILLM) and create a student version by reducing the number of filters in the encoder's convolutional layers by factors of 2, 4, or 8.
- Frozen Decoder: Crucially, they keep the Teacher’s high-quality decoder frozen. The student encoder must learn to produce latent representations that the existing decoder already understands.
- Latent Distillation Loss: To simplify training, they discard the usual reconstruction loss ($L_2$ or MS-SSIM). Instead, the student is trained solely to minimize the Mean Squared Error (MSE) between its output latents ($y_S$) and the teacher’s latents ($y_T$). For hyperprior models, they also match the "hyper-latents" ($z_S$ and $z_T$).
- Resource Efficiency: This "latent matching" is much easier to optimize than the original task, allowing the model to converge faster and with significantly less data.
Datasets
- Training: Subsets of the Vimeo-90k dataset (originally ~89k video clips). The authors tested varying proportions ($\rho$), ranging from 10% down to a tiny 0.1% of the data.
- Evaluation: The classic Kodak dataset (24 images) and the CLIC2020 test split (428 images).
- Teacher Models: Pre-trained on Vimeo-90k (for Factorized Prior) and OpenImages V6 (for the generative MS-ILLM model).
Key Insights and Ideas
- The "Aha" Moment on Data: The most striking finding is how little data is needed. A student encoder trained via KD on just 0.1% of the dataset significantly outperforms a student trained from scratch on the same small slice. The teacher’s latents act as a "dense" supervisory signal that is much more informative than raw pixels.
- Stability through Guidance: Training reduced-width models from scratch is often unstable. Latent distillation provides a stable target, leading to faster convergence and better "statistical fidelity" (the images look more natural and less "blurry").
- Asymmetric Compression: This method enables an asymmetric ecosystem where a very cheap, low-power encoder at the edge can communicate perfectly with a heavy, high-performance decoder in the cloud.
- Simplified Loss works: You don't actually need to calculate the bit-rate or reconstruction error during distillation training; simply mimicking the teacher's bottleneck features is a sufficient proxy for the entire compression task.
Key Results
- Massive Speedups: For the Factorized Prior model, a $\div 8$ reduction in encoder width resulted in a 35x reduction in MACs (Multiply-Accumulate Operations) and a 16x reduction in storage size (from 6.0 MB down to 0.37 MB).
- Superior Quality: Across all tests, the KD-trained students achieved higher PSNR (signal-to-noise ratio) and lower FID (a measure of visual quality/naturalness) than models trained from scratch with the same reduced width.
- Bit-rate Savings: Using Bjøntegaard Delta (BD-Rate) metrics, the KD approach showed significant bit-rate reductions (up to 77% improvement in some cases) compared to the "frozen" baseline trained without distillation.
Limitations and Future Work
- Shared Skeleton: The current method assumes the student and teacher share a similar architectural "skeleton" (just with different widths). It’s unclear how well this would work if the student had a completely different internal structure.
- Encoder-Only: The paper focuses strictly on the encoder. While this is great for IoT upload scenarios, the decoder remains large.
- Future Directions: The authors explicitly mention that their next step is to explore decoder reduction, potentially creating a fully distilled, end-to-end lightweight pipeline. They also aim to test the method on even more complex generative compression models.
Summary updated