Pruned neural networks can lose 70–95% of their weights with little accuracy loss, yet they rarely run faster. The hardware still has to store where each surviving weight sits, fetch those positions, and keep its processing units busy despite uneven work. This proposal removes that bookkeeping on a reconfigurable chip (an FPGA, whose wiring can be reprogrammed). After pruning and quantization, each layer's surviving weights are fixed constants. We compile every layer into a circuit in which each nonzero weight is a small hardwired constant multiplier and each zero is simply a missing wire. Such circuits are too large to hold a whole network at once. So the chip keeps only one or two layers resident and loads the next layer's circuit into a reserved region while the current layer computes. This loading is called dynamic partial reconfiguration: replacing part of the chip's configuration while the rest keeps running.
The claim is narrow and testable. When per-layer work is large enough to hide the time it takes to swap a layer in, weight-specialized circuits beat the best static sparse accelerator on the same chip in throughput per watt. The reason is that they pay nothing for sparsity metadata or for fetching weights from memory. Below that point, they lose. The research question is where that crossover lies and whether pruning can be trained to land on the winning side of it.
What exists and what is missing
Each ingredient exists separately. AH-CNN loads deep CNN layers into reconfigurable regions of a Zynq-7000 only for hard inputs. It uses ordinary vendor-built partial bitstreams (the configuration files that reprogram a chip region), suffers 38–42 ms swaps that force batching, and does not exploit weight sparsity. FOS supplies the runtime machinery: standardized slots, bitstream relocation between slots, and 3.8–6.7 ms module swaps. It also supplies a warning. Its fixed management logic (the "shell") takes 11–13% of a ZCU102 but up to 80% of an Ultra-96, which constrains which devices are viable. AS-Overlays supplies the synthesis method: it assembles circuits from pre-placed netlist blocks with RapidWright and cuts generation from over 6 minutes to about 2 seconds. It does this only for static, regular kernels such as matrix multiply.
On the sparsity side, two static designs show the alternatives. Spartus handles unstructured sparsity on a fixed FPGA design with a column-balanced compressed format and balanced dropout, so every processing unit sees equal work. Hardware-Aware Pruning takes the opposite route and only prunes channels in multiples of the systolic array width of 32, because anything finer gives zero speedup on a fixed array. BitLogic shows that networks compiled directly into FPGA lookup tables reach nanosecond latency, but only for small feedforward models that fit entirely on chip.
The research file's own gap analysis lists the obvious combination, "use netlist stitching to make sparse partial bitstreams," as unexplored (Gaps 2 and 4). We differ from that framing in two ways. First, we hardwire weight values, not just sparsity patterns, so the index metadata is eliminated rather than handled efficiently. Second, we treat the swap-hiding condition as a training-time constraint on how sparse each layer must be, instead of assuming the hardware can be fit to whatever the pruner produces.
Core Claim
Let $R$ be the time to load one layer's partial bitstream. Let $t_s$ and $t_h$ be the time for one input to pass through a layer on a static sparse engine and on the hardwired circuit, respectively. Let $B$ be the number of inputs, or image tiles of a video stream, processed per layer residency. With two alternating slots, one computing while the other loads, the swap is fully hidden when $B\,t_h \ge R$. The hardwired design beats the static engine overall when
$$B\,(t_s - t_h) > \max(0,\; R - B\,t_h).$$
Both sides come from simple accounting: the left is the per-residency time saved by specialization, and the right is the swap time left exposed. We expect $t_s / t_h$ to grow with sparsity. A static engine still streams indices and weight values from memory, while the hardwired circuit has neither. The hardwired area, however, scales with the number of nonzeros times the cost of each constant multiplier. Area therefore bounds how dense a layer can be and still fit one slot. The combined condition defines a region of sparsity, quantization bit-width, and work per layer where the idea wins. Mapping that region is the main empirical result.
Research Program
The first component is a weight-specialized macro library. Under 4-bit quantization, a weight takes one of 15 nonzero values. For each value we pre-synthesize and pre-place a constant-multiplier macro (shift-and-add logic), together with adder-tree and line-buffer macros for convolution windows. Following AS-Overlays, a layer's circuit is then stitched from these macros with RapidWright instead of running full vendor place-and-route. This matters at compile time: it makes building one bitstream per layer, per model, and per pruning iteration cheap enough to put real hardware cost inside the pruning loop. It does not by itself reduce runtime swap latency. Interfaces follow the relocatable-slot conventions of FOS and Maxeler dynamic integration, so one stitched layer can be loaded into either slot.
The second component is slot-fitting pruning. Hardware-Aware Pruning constrains channel counts to multiples of 32. We replace that constraint with a per-layer area budget equal to the capacity of one reconfigurable slot, using a cost model fit to the measured area of each macro. Different layers end up at different sparsity levels. This gives the non-uniform sparsity that N:M training identifies as open and that fixed 2:4 hardware cannot provide. The pruner also has a fallback: it can split an oversized layer across two consecutive residencies, at the cost of one extra swap, which enters the objective through $R$.
The third component is a scheduler. In the spirit of CMSwitch, which keeps mode-switching overhead at 3–5% of execution by planning around it, the scheduler picks $B$ (batch size or tiles per stream window) and the slot assignment so the condition above holds. Where it cannot hold, the scheduler reports this rather than hiding it.
Baselines
The naive reference point is the dense quantized network on a dense systolic engine built on the same ZCU102. It uses identical bit-width, clock target, and memory interface, and is tuned with the same synthesis effort. Every speedup is reported against it.
The strongest competitor is a static unstructured sparse engine in the style of Spartus. We reimplement its column-balanced compressed storage and balanced pruning for convolution on the same board, with the same total fabric (it gets the area our two slots plus shell would occupy) and the same tuning budget. It is the strongest competitor because it already exploits unstructured sparsity on an FPGA without any reconfiguration. Spartus reports 9.4 TOp/s effective batch-1 throughput and 1.1 TOp/J at up to 96% weight sparsity on TIMIT. Its edge variant reports 77.3 GOp/s and 33.6 GOp/s/W. Those figures are for LSTMs on different hardware, so they calibrate our reimplementation rather than serve as a target. Beating this baseline means at least 1.5× higher throughput per watt at matched accuracy on at least one realistic workload, with the swap time included.
The structured competitor is Hardware-Aware Pruning, rerun on our dense engine with its multiple-of-32 rule. Its reported result is a 1.9× speedup, from 85.25 ms to about 45.7 ms per frame, with about 5× parameter compression for Deeplabv3+ on SUN colonoscopy video on an Intel Arria 10. Because that is a different chip, we compare against our rerun, not the quoted number. Beating it means matching its Dice score with lower per-frame latency.
The reconfiguration reference is AH-CNN. It reports 86.27% CIFAR-10 and 94.62% SVHN accuracy, about 408 images per second on SVHN, and 38–42 ms swaps. We compare throughput at equal or better accuracy on the same datasets.
Three ablations isolate the key ingredient. The first keeps reconfiguration but loads a generic sparse engine with index memory instead of a hardwired circuit; this tests whether specialization, not swapping, drives the gain. The second uses hardwired circuits with uniform per-layer sparsity; this tests the slot-fitting pruner. The third disables the alternating slots; this measures how much swap time is actually hidden.
For GPU context we cite Sputnik, which reports a 3.58× geometric-mean speedup over cuSPARSE on V100. We measure energy per inference for the same pruned model on an available GPU.
Why not just…?
Why not just use a static sparse engine like Spartus? This is the real threat, and we take it seriously. If the work per layer is too small to hide a 3.8–6.7 ms swap, the static engine wins outright. That will likely be the case for batch-1 inference on 32×32 CIFAR images. Our claim is limited to regimes with large $B\,t_h$, such as high-resolution video segmentation and batched classification. The crossover sweep over sparsity, bit-width, and $B$ settles where, if anywhere, the hardwired design wins. If it never wins with swap time included, that negative result is itself the finding.
Why not just prune channels in hardware-sized blocks? Channel pruning needs no special hardware, and Hardware-Aware Pruning gets 1.9× this way. That paper also shows the cost of the approach. Residual connections tie channel counts across many layers, which limits how far pruning can go, and coarse blocks lose accuracy faster than unstructured pruning, as N:M training notes. We compare both on the colonoscopy accuracy–latency curve. If channel pruning dominates at every point, the extra complexity of our approach is not justified.
Why not just hardwire the whole network statically, as BitLogic does? When the model fits on chip, that is strictly better: no swaps at all. We include a fully static hardwired version wherever it fits. The question is model size. BitLogic reaches 72.3% on CIFAR-10 with small feedforward models and reports that convolutional variants underperform. We measure the largest model that fits statically against the accuracy that time-multiplexing makes reachable.
Why not just use 2:4 sparse tensor cores or Sputnik on a GPU? These are turnkey options, and 2:4 sparsity reaches 77.0% top-1 on ImageNet versus 77.3% dense. However, the 2:4 pattern is fixed in silicon and cannot match non-uniform layer sensitivity, and the energy budget of a GPU is out of reach at the edge. We report energy per frame so readers can judge that trade-off directly.
Proposed Experiments
All FPGA designs run on a ZCU102, where the FOS shell overhead is moderate. We repeat one configuration on an Ultra-96 to show how the 80% shell cost there affects the result. Workloads are CIFAR-10 and SVHN with the AH-CNN backbone, a ResNet-18 pruned with the slot-fitting rule and checked for accuracy only on ImageNet-1K, and Deeplabv3+ on SUN-SEG-Easy and SUN-SEG-Hard. Metrics are accuracy or Dice score, frames per second, energy per frame, measured swap time, the fraction of swap time exposed rather than hidden, and the hardwired area per nonzero weight.
The decisive check is the crossover map. For each layer we compare the measured $t_s$, $t_h$, and $R$ against the inequality above. If the predicted boundary matches where measured throughput flips, the cost model is validated. If it does not, the mismatch shows which term (area, routing, or memory contention between slots) breaks the accounting.
A second check isolates the source of the gain. The index-based reconfigurable ablation should lose most of the advantage. If it does not, the gain comes from swapping and memory locality rather than from removing sparsity metadata, and the core claim fails.
A third check compares slot-fitting pruning with uniform pruning at matched average sparsity. It should improve accuracy, the number of swaps, or both.
References
- A Novel Design of Adaptive and Hierarchical Convolutional Neural Networks using Partial Reconfiguration on FPGA
- FOS: A Modular FPGA Operating System for Dynamic Workloads
- Automatic Generation of Application-Specific FPGA Overlays with RapidWright
- Spartus: A 9.4 TOp/s FPGA-Based LSTM Accelerator Exploiting Spatio-Temporal Sparsity
- Hardware-Aware Pruning for FPGA Deep Learning Accelerators
- BitLogic: Training Framework for Gradient-Based FPGA-Native Neural Networks
- Enabling Dynamic System Integration on Maxeler HLS Platforms
- Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
- Be CIM or Be Memory: A Dual-mode-aware DNN Compiler for CIM Accelerators
- Sparse GPU Kernels for Deep Learning