Modern deep learning models are often too large to run on mobile or IoT devices, yet sending raw data to the cloud for processing raises concerns about privacy, latency, and bandwidth. This has led to the emergence of collaborative intelligence, a paradigm where a neural network is "split" into two parts: a lightweight portion that runs locally on the edge device and a compute-heavy portion that runs on a powerful server.
The core challenge of this research direction is identifying the optimal partition point—the specific layer where the network should be divided to minimize the total cost of computation and communication.
The Split Point Trade-off
At first glance, it might seem that splitting a network as early as possible is best for the edge device, as it minimizes local computation. However, as Deniz Gunduz and Mikolaj Jankowski explain in Deep Joint Transmission-Recognition for Power-Constrained IoT Devices, the choice of split point involves a fundamental tension between local energy expenditure and transmission requirements.
Early layers in a convolutional neural network (CNN) often act as feature extractors that expand the dimensionality of the input. If you split the network too early, the "smashed data" (the intermediate activations) can actually be significantly larger than the original input image. Weiqing Ren and colleagues note in A Survey on Collaborative DNN Inference for Edge Intelligence that identifying layers with minimal data footprints is critical; otherwise, offloading computation might actually increase total latency due to the massive volume of data being transferred.
Finding the Natural Bottlenecks
To avoid the "data expansion trap," researchers look for architectural bottlenecks. Hyomin Choi and Ivan V. Bajic suggest in Deep Feature Compression for Collaborative Object Detection that pooling layers are ideal split points because they naturally reduce data dimensionality.
By placing the cut immediately after a downsampling operation, such as max-pooling, the system can leverage the network's inherent compression. Wenqi Shi and Sheng Zhou demonstrate in Improving Device-Edge Cooperative Inference of Deep Learning via 2-Step Pruning that this strategy can reduce transmission workloads by over 25x.
Compressing the Latent Space
Even at a bottleneck, the raw activations may still be too large for constrained wireless links. This has led to a variety of techniques designed to compress the intermediate representations:
- Spectral Sparsity: In FourierCompress, Jian Ma and Xinchen Lyu exploit the fact that early-layer activations are often spatially smooth. By transforming these features into the frequency domain using FFT, they can achieve high compression ratios with minimal accuracy loss.
- Dimensionality Reduction: Juliano S. Assine and Eduardo Valle explore using PCA to find a "sweet spot" in the middle of the network where representations are compact enough to transmit but robust enough to survive compression, as detailed in Compressing Representations for Embedded Deep Learning.
- Quantization: Frameworks like QPART, developed by Xiangchen Li and Bo Ji, jointly optimize the split point and the bit-precision of each layer. This allows the system to use aggressive 8-bit or lower quantization on the transmitted features to slash bandwidth costs (QPART: Adaptive Model Quantization and Dynamic Workload Balancing for Accuracy-aware Edge Inference).
Adapting to Heterogeneity
The "optimal" split is rarely static. It depends on the current network bandwidth, the battery level of the device, and the specific hardware accelerators available.
For instance, Dayoung Chun and Hyun Kim show in CP-CNN that on heterogeneous embedded systems (like those in autonomous cars), the split point should be chosen to balance the workload between a GPU and a specialized Deep Learning Accelerator (DLA) to maximize throughput.
Similarly, in the context of Split Learning (distributed training), Wei Ni and Xianhao Chen propose HASFL, which dynamically adjusts split points to prevent "straggler" devices with slower processors from delaying the entire training process (HASFL: Heterogeneity-aware Split Federated Learning over Edge Computing Systems).
Why It Works
Collaborative partitioning works because neural networks are not monolithic blocks of computation; they are hierarchies of feature transformations. By viewing the network as a sequence of data-reduction steps, researchers can treat the model itself as a communication protocol.
The goal is to find the layer where the "value" of the features (in terms of task accuracy) is highest relative to their "cost" (in terms of bits and Joules). As Alon Harell and Ivan V. Bajic demonstrate in their work on Rate-Distortion Optimization for Transformer Inference, this is essentially a problem of finding the most efficient way to describe the input data for a specific downstream task.