Distributed inference frameworks—often called "collaborative intelligence" or "split computing"—aim to divide the computational workload of a Deep Neural Network (DNN) between a resource-constrained edge device and a powerful cloud server. By executing the initial layers locally on the device and offloading the remaining deep layers to the cloud, split inference promises to minimize local energy consumption and protect data privacy.
However, this paradigm faces a severe, counter-intuitive obstacle: intermediate feature expansion.
The Paradox of Split Inference
Intuitively, one might assume that processing an image through a few neural network layers would reduce its size, distilling the raw pixels into a compact semantic representation. In reality, the opposite frequently occurs in the early layers of modern DNNs. This phenomenon, known as data amplification, means that the intermediate feature maps can be significantly larger in data volume than the original raw input.
As identified by Choi et al. and Eshratifar et al., if a model is partitioned at a layer experiencing feature expansion, transmitting the resulting tensor across a wireless network requires more bandwidth and introduces higher latency than simply sending the uncompressed raw input. This communication bottleneck completely negates the performance benefits of offloading computation, a challenge thoroughly documented across vehicular and edge networks by Shao et al. and Ren et al..
Raw Input (e.g., 3 Channels) ---> [ Early DNN Layers ] ---> Expanded Feature Map (e.g., 128 Channels)
|
[ Massive Network Bottleneck ]
|
v
[ Deep Cloud Layers ]
To understand why this expansion happens, consider how convolutional networks and vision transformers operate. Early layers apply dozens or hundreds of filters to extract low-level spatial features such as edges, textures, and color gradients. While the spatial dimensions (width and height) may decrease slightly via pooling or strided convolutions, the channel dimension explodes (e.g., from 3 RGB channels to 64 or 128 feature channels). Frickenstein et al. note that these expansion operations dramatically increase data movement costs. Because standard models are trained solely for accuracy rather than communication efficiency, these high-dimensional tensors are highly dense and redundant.
To make split inference viable, researchers have developed mechanisms to compress, prune, and restructure intermediate feature maps precisely at the network's split point.
Structural Bottlenecks and Learned Compressibility
One primary strategy for mitigating feature expansion is forcing the network to learn a compact intermediate representation. Rather than relying on standard architectures, researchers insert specialized, low-dimensional "bottlenecks" at the partition boundary.
- Architectural Bottlenecks: Eshratifar et al. introduced the "Butterfly Unit," which uses a trainable $1 \times 1$ convolutional layer to aggressively downsample feature channels before transmission. Similarly, Datta et al. demonstrated that inserting and training tiny bottleneck units while keeping the rest of a pre-trained model frozen can reduce the required bit-rate up to 392x, requiring less than 1% of the model's parameters to be retrained.
- Automated Architecture Design: Instead of manually designing these bottlenecks, single-author work by Odema demonstrated that neural architecture search (LENS) can automatically discover "splittable" network designs that naturally minimize transmission delay under varying wireless conditions.
- Compressibility Losses: Rather than altering the physical structure, networks can be incentivized mathematically to produce compressible outputs. Bajic et al. introduced a differentiable "compressibility loss" into the multi-task training objective. This forces the network to output features with lower entropy, making them significantly easier for downstream codecs to compress without degrading task accuracy.
Exploiting Redundancy Across Dimensions
Intermediate feature tensors contain massive amounts of mutual information distributed across space, time, and channels. Specialized compression frameworks exploit these specific dimensions to avoid transmitting redundant data.
Channel Redundancy
Not all feature channels contribute equally to a model's final prediction. In the "Back-and-Forth" (BaF) prediction scheme developed by Choi et al., the edge device transmits only a highly correlated subset of the feature channels. The cloud server then utilizes a small, specialized neural network to reconstruct the missing channels on the fly, bypassing the need to transmit the full tensor volume.
Temporal Redundancy
In video processing and surveillance applications, consecutive frames yield highly repetitive feature maps. Iino et al. developed Inter-Feature-Map Differential Coding (IFMDC), which calculates and encodes only the pixel-level differences between consecutive feature maps over time, achieving high compression ratios comparable to traditional video codecs while preserving downstream object detection accuracy.
Sparsity
Features often contain many near-zero or non-essential activations. Qin et al. framed the problem of distributed feature sharing as a "weight pruning" task within their DISCO framework, systematically eliminating the dense connectivity requirements of within-layer parallel processing to reduce communication overhead by up to 90%.
Coding for Machines vs. Humans
Traditional data compression algorithms (such as JPEG, PNG, or H.264) are designed around the limitations of human perception, discarding high-frequency details that the human eye cannot easily detect. In split inference, however, the consumer of the data is not a human, but the remaining layers of a machine learning model. This shift fundamentally alters the rules of rate-distortion optimization.
Harell et al. provided theoretical and empirical proof that optimizing codecs specifically to preserve deeper, task-specific features yields vastly superior rate-distortion performance for machine tasks compared to standard image compression.
However, compressing deeper layers introduces unique challenges. In a subsequent study focusing on Vision Transformers, Harell et al. uncovered a counter-intuitive phenomenon: deeper transformer layers can actually be harder to compress than earlier ones, despite having lower theoretical entropy. This occurs because the underlying feature distributions become highly complex and non-linear as data moves deeper into the network, requiring specialized transformer-based entropy models to accurately capture and compress them.
Traditional Compression: Raw Image ----> [ Human Eye Optimization ] ----> Human Viewer
Feature Compression: Raw Image ----> [ Edge Layers ] ----> [ Machine Optimization ] ----> [ Cloud Layers ]
To handle the structural diversity of modern foundation models, Li et al. developed a universal feature coding framework (DT-UFC). Recognizing that different large models (such as LLaMA3 or DINOv2) produce wildly heterogeneous, "peaky" activation distributions, the authors designed a universal transformation that maps these disparate feature spaces into a unified, balanced distribution. This allows a single codec to efficiently compress intermediate features across entirely different model architectures without requiring model-specific retraining.
Key Foundational Work
Towards Collaborative Intelligence Friendly Architectures for Deep Learning (2019) — contributed 2 concept elements to this direction.
BottleNet: A Deep Learning Architecture for Intelligent Mobile Cloud Computing Services (2019) — contributed 2 concept elements to this direction.
Improving Device-Edge Cooperative Inference of Deep Learning via 2-Step Pruning (2019) — contributed 1 concept elements to this direction.