Optimal Partitioning for Collaborative Neural Network Inference

This direction is merged into another direction: distributed model partitioning. Its last version stays readable and citable here.

Version 1 of 140 papers54 conceptsfrozen 2026-10-04

Modern deep learning models are often too large to run on mobile or IoT devices, yet sending raw data to the cloud for processing raises concerns about privacy, latency, and bandwidth. This has led to the emergence of collaborative intelligence, a paradigm where a neural network is "split" into two parts: a lightweight portion that runs locally on the edge device and a compute-heavy portion that runs on a powerful server.

The core challenge of this research direction is identifying the optimal partition point—the specific layer where the network should be divided to minimize the total cost of computation and communication.

The Split Point Trade-off

At first glance, it might seem that splitting a network as early as possible is best for the edge device, as it minimizes local computation. However, as Deniz Gunduz and Mikolaj Jankowski explain in Deep Joint Transmission-Recognition for Power-Constrained IoT Devices, the choice of split point involves a fundamental tension between local energy expenditure and transmission requirements.

Early layers in a convolutional neural network (CNN) often act as feature extractors that expand the dimensionality of the input. If you split the network too early, the "smashed data" (the intermediate activations) can actually be significantly larger than the original input image. Weiqing Ren and colleagues note in A Survey on Collaborative DNN Inference for Edge Intelligence that identifying layers with minimal data footprints is critical; otherwise, offloading computation might actually increase total latency due to the massive volume of data being transferred.

Finding the Natural Bottlenecks

To avoid the "data expansion trap," researchers look for architectural bottlenecks. Hyomin Choi and Ivan V. Bajic suggest in Deep Feature Compression for Collaborative Object Detection that pooling layers are ideal split points because they naturally reduce data dimensionality.

By placing the cut immediately after a downsampling operation, such as max-pooling, the system can leverage the network's inherent compression. Wenqi Shi and Sheng Zhou demonstrate in Improving Device-Edge Cooperative Inference of Deep Learning via 2-Step Pruning that this strategy can reduce transmission workloads by over 25x.

Compressing the Latent Space

Even at a bottleneck, the raw activations may still be too large for constrained wireless links. This has led to a variety of techniques designed to compress the intermediate representations:

Adapting to Heterogeneity

The "optimal" split is rarely static. It depends on the current network bandwidth, the battery level of the device, and the specific hardware accelerators available.

For instance, Dayoung Chun and Hyun Kim show in CP-CNN that on heterogeneous embedded systems (like those in autonomous cars), the split point should be chosen to balance the workload between a GPU and a specialized Deep Learning Accelerator (DLA) to maximize throughput.

Similarly, in the context of Split Learning (distributed training), Wei Ni and Xianhao Chen propose HASFL, which dynamically adjusts split points to prevent "straggler" devices with slower processors from delaying the entire training process (HASFL: Heterogeneity-aware Split Federated Learning over Edge Computing Systems).

Why It Works

Collaborative partitioning works because neural networks are not monolithic blocks of computation; they are hierarchies of feature transformations. By viewing the network as a sequence of data-reduction steps, researchers can treat the model itself as a communication protocol.

The goal is to find the layer where the "value" of the features (in terms of task accuracy) is highest relative to their "cost" (in terms of bits and Joules). As Alon Harell and Ivan V. Bajic demonstrate in their work on Rate-Distortion Optimization for Transformer Inference, this is essentially a problem of finding the most efficient way to describe the input data for a specific downstream task.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_optimal_partitioning_for_collaborative_n_092d495a,
  title = {Optimal Partitioning for Collaborative Neural Network Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimal-partitioning-for-collaborative-neural-network-inference/rdr_092d495ae48742d3b04c7b3dba785d5e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_optimal_partitioning_for_collaborative_n_092d495a_v1,
  title = {Optimal Partitioning for Collaborative Neural Network Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimal-partitioning-for-collaborative-neural-network-inference/rdr_092d495ae48742d3b04c7b3dba785d5e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}