Coded Deep Learning: Framework and Algorithm

En-Hui Yang, Shayan Mohajer Hamidi

Abstract

Deep Neural Networks (DNNs) have achieved remarkable performance across various domains but remain computationally expensive and memory-intensive, hindering deployment on edge devices and efficient distributed training. This paper presents Coded Deep Learning (CDL), a novel framework that integrates information-theoretic principles directly into the neural network training process. By utilizing trainable probabilistic quantizers and an entropy-constrained loss function, CDL enables models to learn inherently compressible representations. Experimental results demonstrate that CDL and its relaxed variant (R-CDL) consistently outperform state-of-the-art quantization methods, often achieving higher accuracy than full-precision models while operating at extremely low bit-widths.

1. Introduction

The deployment of modern DNNs is frequently limited by the "memory wall" and communication bottlenecks. Standard approaches to model compression, such as Quantization-Aware Training (QAT), typically rely on the Straight-Through Estimator (STE) to bypass the non-differentiability of quantization functions. However, the STE is a coarse approximation that often leads to sub-optimal convergence. Furthermore, traditional compression methods often treat quantization as a post-hoc or secondary step rather than a fundamental constraint of the learning process.

Coded Deep Learning (CDL) addresses these challenges by "baking in" compressibility from the start. The framework treats quantization as a stochastic process and optimizes for both task accuracy and the Shannon entropy of weights and activations. This approach ensures that the resulting models are not just quantized, but are structured for maximum compressibility via entropy coding.

2. Methodology

The CDL framework introduces two primary variants: CDL, where training occurs directly on quantized values to reduce training-time complexity, and Relaxed CDL (R-CDL), which utilizes full-precision values during training while optimizing for future quantization to maximize final accuracy.

2.1 Probabilistic and Soft Deterministic Quantizers

To overcome the gradient gap inherent in standard quantization, the authors introduce two key innovations:

  • Probabilistic Quantizers ($Q_p$): Instead of deterministic rounding, CDL employs a trainable Conditional Probability Mass Function (CPMF). Quantization is treated as a stochastic process governed by a parameter $\alpha$, which controls the transition from continuous to discrete states.
  • Soft Deterministic Proxy ($Q_d$): To ensure differentiability without the inaccuracies of the STE, the authors derived a "soft" analytic formula for the quantizer. This proxy provides smooth, well-defined gradients during backpropagation. As training progresses and $\alpha$ increases, $Q_d$ converges to a standard uniform quantizer, providing a stable bridge between continuous optimization and discrete deployment.

2.2 Joint Entropy Optimization

Unlike standard training objectives, the CDL loss function incorporates the Shannon Entropy of the weights ($H(w)$) and activations ($H(x)$). By minimizing entropy during training, the model is incentivized to find "low-entropy" solutions—weight distributions and activation patterns that can be represented with fewer bits without sacrificing predictive power. This makes the models inherently compatible with subsequent compression algorithms like Huffman coding.

3. Experimental Evaluation

The framework was evaluated on standard computer vision benchmarks to test its robustness across different architectures and scales:

  • CIFAR-100: Tested using ResNet (20, 44, 56, 110), VGG-13, and Wide-ResNet-28-10.
  • ImageNet: Tested using ResNet-18 and ResNet-34 to evaluate performance on large-scale classification tasks.

4. Key Results

The CDL framework demonstrates significant improvements over existing state-of-the-art (SOTA) methods such as LSQ, PACT, and LQ-Nets, particularly in low-bit regimes.

  • Surpassing Full Precision: In a notable result on ImageNet, an R-CDL trained ResNet-18 achieved 70.55% accuracy using only 2.34 bits per weight. This exceeded the accuracy of the original 32-bit full-precision model (70.5%).
  • Superior Compression-Accuracy Trade-off: R-CDL consistently maintained higher accuracy than competitors when pushed to 2 bits per weight. On ResNet-110, the authors demonstrated over 13-fold compression with negligible loss in accuracy.
  • Communication Efficiency: By constraining the entropy of activations, CDL significantly reduces the data volume required for transmission between GPUs in pipeline-parallel distributed training, addressing a major bottleneck in scaling large models.

5. Discussion and Limitations

While CDL represents a significant advancement, it introduces certain trade-offs. The use of Softmax operations within the quantizers—particularly for activations—can increase computational overhead during training. The authors mitigate this by restricting calculations to "top-k" values, but it remains a consideration for training efficiency. Additionally, while R-CDL offers superior accuracy, it requires full-precision operations during the training phase, whereas standard CDL is more suited for reducing training-time FLOPs.

Future work involves scaling the CDL framework to Transformer architectures and Large Language Models (LLMs), where the benefits of entropy-constrained activations could drastically improve the efficiency of massive-scale distributed training.

6. Conclusion

Coded Deep Learning shifts the paradigm of model compression from post-training heuristics to an integrated, information-theoretic optimization problem. By replacing the crude approximations of the Straight-Through Estimator with differentiable probabilistic proxies and optimizing for entropy, CDL produces models that are "born compressed," achieving state-of-the-art performance in the low-bit regime.

Summary updated

Abstract

The success of deep learning (DL) is often achieved with large models and high complexity during both training and post-training inferences, hindering training in resource-limited settings. To alleviate these issues, this paper introduces a new framework dubbed ``coded deep learning'' (CDL), which integrates information-theoretic coding concepts into the inner workings of DL, to significantly compress model weights and activations, reduce computational complexity at both training and post-training inference stages, and enable efficient model/data parallelism. Specifically, within CDL, (i) we first propose a novel probabilistic method for quantizing both model weights and activations, and its soft differentiable variant which offers an analytic formula for gradient calculation during training; (ii) both the forward and backward passes during training are executed over quantized weights and activations, eliminating most floating-point operations and reducing training complexity; (iii) during training, both weights and activations are entropy constrained so that they are compressible in an information-theoretic sense throughout training, thus reducing communication costs in model/data parallelism; and (iv) the trained model in CDL is by default in a quantized format with compressible quantized weights, reducing post-training inference and storage complexity. Additionally, a variant of CDL, namely relaxed CDL (R-CDL), is presented to further improve the trade-off between validation accuracy and compression though requiring full precision in training with other advantageous features of CDL intact. Extensive empirical results show that CDL and R-CDL outperform the state-of-the-art algorithms in DNN compression in the literature.
BibTeX
@article{yang2025coded,
  author = {En-Hui Yang and Shayan Mohajer Hamidi},
  title = {{Coded Deep Learning: Framework and Algorithm}},
  journal = {IEEE Trans. Information Theory},
  year = {2025},
  doi = {10.1109/tit.2025.3610095},
}
Figure