Deep learning has long operated under a strict paradigm: to build more capable models, we must stack layers deeper or make them wider. However, this architectural philosophy introduces a fundamental hardware challenge during inference. In a traditional deep neural network, Layer $N+1$ cannot begin its computation until Layer $N$ has completely finished and passed its output forward. This strict serial dependency creates a theoretical latency floor that is bounded by the physical clock speeds of modern processors.
While pipeline parallelism can improve throughput for large batches of data, it does little to accelerate a single inference request, leaving hardware units idle while waiting for upstream data. To break through this latency floor, a growing body of research explores concurrent and intra-layer execution. Instead of processing a network as a rigid linear sequence, these techniques restructure models and execution schedules to compute layers, subgraphs, or individual layer components simultaneously across parallel hardware units.
Exploiting Representation Redundancy Across Layers
The most intuitive reason we can execute layers concurrently is that deep models—particularly Transformers—do not always radically transform representations from one layer to the next. Instead, the intermediate layers often act like an assembly line of painters who share a common vocabulary, gradually refining the same underlying representation space. As discovered by Nain et al., these middle layers are highly uniform; they can be reordered, skipped, or even executed in parallel with only a graceful degradation in final model accuracy.
Sequential Execution (Traditional):
[ Input ] ---> [ Layer N ] ---> [ Layer N+1 ] ---> [ Output ]
Concurrent Execution (CQIL / Painters):
+---> [ Layer N ] ---+
[ Input ] -----| |---> [ Average/Aggregate ] ---> [ Output ]
+---> [ Layer N+1 ] ---+
To exploit this property for speed, Zou et al. introduced a method called CQIL (Concurrent Computation of Quasi-Independent Layers). By identifying adjacent layers that receive highly similar inputs, the authors distribute these "quasi-independent" layers across multiple GPUs. Instead of forcing Layer $N+1$ to wait for Layer $N$, both layers receive the exact same input simultaneously. Their outputs are then aggregated, effectively cutting the execution time of those blocks in half with negligible impact on accuracy.
Splitting the Workload Within a Single Layer
When layers cannot be skipped or run in parallel due to strict information dependencies, the alternative is to parallelize the computation within the layer itself. Rather than assigning entire layers to different devices, intra-layer parallelism splits a single layer's matrix multiplications or convolutional kernels across multiple processing units.
This approach is particularly vital for feedback-intensive tasks like Large Language Model (LLM) text generation, where each token depends on the last, making traditional pipelining highly inefficient. To address this, Hong et al. developed DFX, a multi-FPGA appliance that partitions operations within a single transformer layer. By distributing the layer's workload across multiple chips, the system minimizes synchronization overhead and accelerates the sequential generation loop.
This within-layer distribution strategy scales from edge clusters down to specialized silicon:
- Edge Networks: Qin et al. developed DISCO, a framework that splits individual layers across multiple low-power edge devices. To overcome the network communication bottleneck inherent in distributed edge computing, the authors treat the question of "what data to share" as a weight pruning task, reducing communication overhead by up to 90%. Similarly, Baccour et al. parallelize individual layer components across a swarm of IoT devices, which simultaneously accelerates inference and enhances privacy by ensuring no single untrusted device receives enough fine-grained feature map data to reconstruct the original input image.
- Analog Hardware: At the hardware level, Li et al. parallelize convolutional operations across separate analog memristive crossbar arrays. By mapping different kernels to distinct crossbars, the system executes convolutions simultaneously in memory, bypassing the serial processing bottlenecks of traditional digital architectures.
Structural Concurrency via Subnetworks and Branches
Beyond optimizing existing sequential models, researchers are designing architectures and runtimes around parallel branches. The core insight here is that modern hardware can execute multiple small, independent operations simultaneously with virtually the same latency as a single operation.
For instance, Goyal et al. challenged the "deeper is better" paradigm by introducing ParNet (Parallel Networks). Instead of stacking 50 or 100 layers sequentially, ParNet achieves state-of-the-art computer vision accuracy using a depth of just 12 layers by routing features through parallel subnetworks. This design shifts the computational burden from sequential depth to hardware width, allowing multi-chip modules to process the streams concurrently.
A similar logic applies to ensembles and multi-model cascades. Zhao et al. proposed "Divide and Co-training," which splits a single wide network into an ensemble of smaller sub-networks. Executing these smaller models concurrently across separate hardware units yields a better accuracy-efficiency trade-off than running one massive, monolithic model.
Furthermore, modern hardware architectures allow for the parallel execution of multiple small models with negligible latency penalties compared to a single model. This characteristic enables advanced routing mechanisms:
- Model Cascades: Kolawole et al. leverage this hardware capability in Agreement-Based Cascading (ABC), running an ensemble of cheap models simultaneously to handle easy examples and only invoking expensive models when the ensemble disagrees.
- Adversarial Detection: Deng et al. utilize group convolutions and batch matrix multiplications to execute multiple Bayesian ensemble members simultaneously, adding robust adversarial detection capabilities with almost zero latency overhead.
- Multi-Task and Multi-Step Processing: This branching logic extends to multi-task networks—where Markhorst et al. note that executing denoising and classification branches in parallel can significantly reduce edge device latency—as well as text generation, where Wu et al. parallelize reasoning sub-steps through specialized modules to avoid the linear time growth of traditional sequential Chain-of-Thought generation.
Monolithic Sequential Model:
[ Input ] ---> [ Deep, Heavy Layers (100+ Layers) ] ---> [ Output ]
Parallel Branch / Ensemble Model:
+---> [ Subnetwork A ] ---+
[ Input ] -----|---> [ Subnetwork B ] ---|---> [ Combine ] ---> [ Output ]
+---> [ Subnetwork C ] ---+
(Executed simultaneously on multi-core / heterogeneous hardware)
Maximizing Heterogeneous Hardware Utilization
On asymmetric consumer hardware, such as mobile Systems-on-Chip (SoCs) containing both CPUs and GPUs, concurrent execution becomes a scheduling problem. If a model execution graph contains independent branches, a smart runtime can hide the latency of one branch behind another.
Heterogeneous Branch Scheduling (YOLObile / Parallax):
+---> [ Light Branch / Fallback ] ---> [ Mobile CPU ] ---+
[ Graph Split ]| |---> [ Sync ]
+---> [ Heavy Convolutional Task ] ---> [ Mobile GPU ] ---+
This strategy was successfully implemented by Cai et al. in their YOLObile framework. By analyzing a neural network's computation graph, their compilation design routes lighter, non-convolutional operations to the mobile CPU while the GPU handles heavy convolutional workloads simultaneously. This heterogeneous concurrent scheduling effectively hides the CPU's latency behind the GPU's processing time.
To bring these benefits to general models without manual code modifications, Tang et al. developed Parallax. When an edge GPU encounters an unsupported operator, it typically falls back to the CPU sequentially, stalling execution. Parallax solves this by automatically partitioning the neural network graph at runtime, allowing independent subgraphs to execute on idle CPU cores concurrently with the GPU's primary workload.
Whether by restructuring layers to run simultaneously, splitting individual layer workloads across edge nodes, or scheduling independent graph branches across heterogeneous processors, bypassing sequential bottlenecks allows deep learning models to achieve ultra-low latency inference that physical clock speeds would otherwise prevent.
Key Foundational Work
RL-DistPrivacy: Privacy-Aware Distributed Deep Inference for low latency IoT systems (2022) — contributed 2 concept elements to this direction.
Toward Better Accuracy-Efficiency Trade-Offs: Divide and Co-Training (2022) — contributed 2 concept elements to this direction.
Seizure Detection and Prediction by Parallel Memristive Convolutional Neural Networks (2022) — contributed 2 concept elements to this direction.