Lightweight compression of neural network feature tensors for collaborative intelligence

Robert A. Cohen, Hyomin Choi, Ivan V. Bajic

Abstract

In collaborative intelligence applications, part of a deep neural network (DNN) is deployed on a relatively low-complexity device such as a mobile phone or edge device, and the remainder of the DNN is processed where more computing resources are available, such as in the cloud. This paper presents a novel lightweight compression technique designed specifically to code the activations of a split DNN layer, while having a low complexity suitable for edge devices and not requiring any retraining. We also present a modified entropy-constrained quantizer design algorithm optimized for clipped activations. When applied to popular object-detection and classification DNNs, we were able to compress the 32-bit floating point activations down to 0.6 to 0.8 bits, while keeping the loss in accuracy to less than 1%. When compared to HEVC, we found that the lightweight codec consistently provided better inference accuracy, by up to 1.3%. The performance and simplicity of this lightweight compression technique makes it an attractive option for coding a layer's activations in split neural networks for edge/cloud applications.

Introduction: The Bottleneck of Collaborative Intelligence

Collaborative Intelligence (CI) involves splitting a heavy deep neural network (DNN) between an edge device (e.g., a smartphone) and a powerful cloud server. The first few layers are executed locally, and the resulting intermediate feature tensors are transmitted to the cloud for the remaining processing.

However, a significant challenge arises: feature tensors are often much larger and more redundant than the original input images. Transmitting these raw 32-bit floating-point tensors can consume excessive bandwidth, negating the benefits of the split. While standard video codecs like HEVC can be used, they are computationally intensive for edge hardware and optimized for natural images rather than abstract feature maps. Furthermore, many existing compression techniques require "quantization-aware retraining," which is often impractical.

Proposed Methodology

The authors propose a four-stage, post-training compression pipeline designed for maximum efficiency without requiring model retraining.

Overview of the collaborative intelligence pipeline showing the split between mobile/edge device and cloud/compute device.
  1. Clipping: Activations are clamped between a minimum ($c_{min}$) and maximum ($c_{max}$) value. This removes extreme outliers, allowing quantization bins to focus on the most informative data range.
  2. Coarse Scalar Quantization: Clipped values are mapped to a small set of representative levels.
  3. Modified Entropy-Constrained Quantizer (ECQ) Design: This is the core innovation. Standard ECQs minimize Mean Square Quantization Error (MSQE) by pulling levels toward the data's center, which can "shrink" the tensor's dynamic range. The proposed method "pins" the outermost reconstruction levels to the clipping boundaries, preserving the full range of the layer.
  4. Binarization & Entropy Coding: Quantized indices are converted to binary strings via Truncated Unary binarization and compressed using Context-based Adaptive Binary Arithmetic Coding (CABAC), a high-efficiency engine from modern video standards.

Key Insights: Accuracy vs. MSQE

A critical finding of the research is that Mean Square Quantization Error (MSQE) is a deceptive metric for DNN performance at low bitrates. The clipping range that minimizes mathematical error does not always yield the highest task accuracy (e.g., mAP for object detection). For neural networks, preserving the structural range of the data is often more vital than minimizing pixel-wise reconstruction error.

Additionally, by avoiding complex spatial transforms like the Discrete Cosine Transform (DCT) used in video codecs, the proposed method is over 90% less computationally complex than HEVC, making it highly suitable for battery-constrained mobile devices.

Experimental Results

The method was evaluated on COCO 2017 (using YOLOv3) and ImageNet (using VGG-16 and ResNet-50).

Comparison of mAP vs. bitrate for YOLOv3 on COCO 2017, showing the proposed method outperforming HEVC SCC.
  • High Compression Ratios: 32-bit activations were successfully compressed to between 0.6 and 0.8 bits per element.
  • Negligible Accuracy Loss: At these bitrates, the drop in inference accuracy was kept under 1%.
  • Superior to Video Standards: The lightweight codec outperformed the HEVC Screen Content Coding (SCC) extension by up to 1.3% in accuracy at comparable bitrates.
  • Extreme Quantization: For certain models like YOLOv3 and ResNet-50, even 1-bit quantization proved feasible, maintaining reasonable accuracy while drastically reducing data size.

Limitations and Future Work

While effective, the current approach relies on empirical tuning to find optimal clipping ranges. Future research could automate this selection based on layer-specific statistical properties. Additionally, the performance impact of different split points and actual hardware power/latency measurements on mobile CPUs or NPUs remain areas for further investigation.

Summary updated

Abstract

In collaborative intelligence applications, part of a deep neural network (DNN) is deployed on a relatively low-complexity device such as a mobile phone or edge device, and the remainder of the DNN is processed where more computing resources are available, such as in the cloud. This paper presents a novel lightweight compression technique designed specifically to code the activations of a split DNN layer, while having a low complexity suitable for edge devices and not requiring any retraining. We also present a modified entropy-constrained quantizer design algorithm optimized for clipped activations. When applied to popular object-detection and classification DNNs, we were able to compress the 32-bit floating point activations down to 0.6 to 0.8 bits, while keeping the loss in accuracy to less than 1%. When compared to HEVC, we found that the lightweight codec consistently provided better inference accuracy, by up to 1.3%. The performance and simplicity of this lightweight compression technique makes it an attractive option for coding a layer's activations in split neural networks for edge/cloud applications.
BibTeX
@inproceedings{cohen2020lightweight,
  author = {Robert A. Cohen and Hyomin Choi and Ivan V. Bajic},
  title = {{Lightweight compression of neural network feature tensors for collaborative intelligence}},
  booktitle = {ICME},
  year = {2020},
  url = {https://openreview.net/forum?id=mXFpADf9g8},
}
Figure