Keeping Restoration Activations on the Wires: An Exposed-Datapath Co-Processor for Activation-Free Networks

Image restoration networks, the models that deblur, denoise or upscale photos, are expensive on small devices for a reason that differs from classifiers. They keep the image at full resolution through every layer. The main cost is therefore moving large intermediate feature maps in and out of memory, often more than the arithmetic itself. NAFNet, from Simple Baselines for Image Restoration, showed that such networks can drop sigmoids and GELUs entirely. Its "SimpleGate" multiplies two halves of a feature map together, which makes each block almost purely multiplies and adds. It reaches 33.69 dB on GoPro deblurring and 40.30 dB on SIDD denoising at a fraction of prior compute. The block, though, is a long chain of small elementwise steps (normalize, 1×1 conv, depthwise 3×3 conv, gate, channel scale, residual add). On an ordinary accelerator, each step writes its result to a register file or scratchpad and reads it back.

An exposed-datapath processor is a CPU whose instructions are just "move this value from that unit's output to this unit's input," with computation firing when data arrives. This design lets the compiler send a result directly from one arithmetic unit to the next without storing it anywhere (Data Intensive Computing: From Modeling to Implementation). We propose to build and measure a small co-processor of this kind, a transport-triggered architecture (TTA). Its internal wiring is derived from the dataflow of an activation-free restoration block. It streams image rows through an entire NAFNet block so that almost no intermediate value touches a register file or SRAM, and it stays programmable across restoration tasks.

Core Claim

For activation-free restoration blocks, most intra-block data movement can be turned into direct unit-to-unit transports scheduled at compile time. This is possible only if the programmable interconnect is pruned to the block's dataflow graph. Otherwise the "flexibility tax" that BrainTTA measured eats the savings. That paper found that in a fabricated TTA neural SoC, the buses and multiplexers were the second-largest energy consumer after arithmetic. Our claim is quantitative and falsifiable. On the same fixed-point NAFNet-style model, the dataflow-shaped TTA should use substantially less energy per output pixel than an otherwise identical TTA that routes intermediates through its register file. It should also retain enough programmability to run deblurring, denoising and super-resolution variants without re-synthesis.

This idea differs from the closest existing work in specific ways. BrainTTA targets classification convolutions with vector multiply-accumulate reduction trees and uniform per-layer precision. It has no wide elementwise multiplier for gating and no attempt to fuse a whole block. The TTA vision array shows neighbour-register sharing for single 3×3 stencils at 0.17 nJ/pixel, but only for shallow hand-written kernels such as local binary patterns. AEx prunes an oversized TTA from C profiles. It has only been tested on scalar CHStone kernels, and it prunes functional units and ports rather than generating interconnect from a tensor dataflow graph. We reuse the TTA template and the AEx expand-and-prune strategy as method. Our contribution is the target workload, the fusion schedule, and the interconnect generation.

Research Program

The work has three parts: a hardware-friendly model, a compiler schedule, and a synthesized processor.

Model. NAFNet is not quite activation-free in hardware terms, and we treat that honestly. Two operations break simple streaming.

The first is LayerNorm, which needs a per-pixel mean and variance across channels plus a reciprocal square root. We will compare two options. One keeps LayerNorm and adds a small channel-reduction unit with a table-based reciprocal square root. The other retrains the network with normalization folded into the adjacent convolutions. We report the PSNR cost of each.

The second is Simplified Channel Attention, which pools over the entire image before scaling channels. This forces either two passes over the feature map or storing it all. We will retrain with a tile-local pooling window of a few dozen rows, which is compatible with a line buffer, and measure the quality loss on GoPro and SIDD.

All models are trained with quantization-aware training to 8-bit weights and activations. SimpleGate multiplies two activations, which doubles dynamic range, so we insert a per-block requantization shift. We report its effect separately, since the literature on low-bit restoration (BSRN, CADyQ) warns that edges and high frequencies are the first casualties of aggressive quantization.

Schedule. We write the block as a row-streamed, software-pipelined loop. A few input rows sit in a line buffer, and each output row of the block is produced before the next input row arrives. The two halves from the expanding 1×1 convolution go straight to the operand and trigger ports of an elementwise multiplier. Depthwise-conv taps come from neighbour-style registers in the manner of the vision array. The residual add receives the skip value from a short delay FIFO rather than from memory. Only block inputs, outputs, and the UNet skip connections between encoder and decoder go to SRAM or DRAM. We will instrument the schedule to report, for every intermediate value, whether it travelled by bypass, register file, or memory.

Processor. We start from an oversized TTA with wide integer vector multiply-accumulate units, an elementwise vector multiplier, the reduction unit, and a full crossbar. AEx-style profiling is extended in one respect. Bus-to-port connections are pruned using transport counts from the compiled block schedules of a training set of network variants, rather than from functional-unit usage alone. Instruction supply uses a loop buffer or software-managed instruction register file, as in the LoTTA cores, since the block loop body is small and repeated millions of times per frame. We evaluate with the TTA Co-design Environment's cycle-accurate simulator. Energy comes from 28 nm synthesis with activity-annotated power estimation, which is the node AEx and the LoTTA cores report. We use a standard per-byte DRAM energy model for off-chip traffic.

Baselines

The naive reference is the same quantized models compiled layer by layer for the unpruned TTA. Every layer's output goes to SRAM, and off-chip DRAM is used when the feature map exceeds on-chip capacity. This is what a straightforward port looks like. It anchors how much of the gain comes from fusion at all.

The key ablation isolates the proposal's central ingredient. It uses the identical fused row-streaming schedule on a TTA with the same functional units. The only change is that intermediates are routed through a register file instead of direct bypass, with the interconnect sized to match. If this ablation reaches nearly the same energy as the full design, then fusion alone explains the result and exposed bypass adds nothing. A second ablation keeps bypass but uses the unpruned crossbar. This measures how much of BrainTTA's flexibility tax the dataflow-shaped interconnect recovers.

The strongest programmable competitor is BrainTTA itself. Its 8-bit vector multiply-accumulate unit reports 405 fJ/op and 77 GOPS in 22 nm (BrainTTA). No restoration workload has been run on it. We will re-create its datapath organization in our flow, with reduction trees, a register file, and a loop buffer, at the same node and voltage. We add the minimum elementwise support needed to run SimpleGate and compile the same models with the best fused schedule it can express. Beating it means lower measured energy per output pixel at equal PSNR and equal or better throughput on 720p frames. Energy per operation is not a fair metric here, because our design deliberately removes operations' data-movement overhead. We know of no reported result on Lacuna for NAFNet, or any restoration network, on a TTA or comparable edge ASIC. We therefore cannot quote a prior energy-per-pixel number for this workload.

The fixed-function ceiling is a Vitis-style HLS implementation of the same fused NAFNet block with the same line buffers. We synthesize it at the same node and give it the same engineering effort budget as our schedule, measured in person-weeks of tuning. AEx reported that even hand-tuned TTAs remained 1.64x slower than Vitis HLS on AES (AEx). We treat landing within a comparable margin on energy per pixel, while running unseen network variants the fixed design cannot run, as success.

For quality, we use the full-precision NAFNet numbers as the ceiling: 33.69 dB on GoPro and 40.30 dB on SIDD (Simple Baselines for Image Restoration). For energy at low quality, the reference is MuLUT. It reports about 278 µJ per HD image on a Snapdragon 888 for super-resolution (MuLUT). We will run its released tables through our energy model for the SR comparison.

Why not just…?

Why not just build a fixed-function accelerator for the NAFNet block? This is the most serious threat. For one frozen network, a hardwired pipeline with line buffers will probably be faster and possibly more efficient, as the AEx comparison suggests. Our answer rests on programmability, and we test it directly rather than assert it. The pruned TTA must run deblurring, denoising and SR variants with different widths, depths and block counts, including variants withheld from the pruning step. The fixed design cannot run them without re-synthesis. If the energy gap to fixed-function exceeds roughly the 1.64x AEx found and generalization to held-out variants fails, then the fixed-function design wins and we will say so.

Why not just fuse the block on an existing accelerator and keep intermediates in a local scratchpad? Operator fusion is standard, and it captures the off-chip savings. The register-file-routed ablation is exactly this alternative, run on the same processor. What remains open is whether the last level, removing register-file and scratchpad accesses for short-lived values, matters once DRAM traffic is gone. Dally's embedded-processor analysis suggests it does, because multi-ported register files and operand delivery dominate core energy. BrainTTA's interconnect finding cuts the other way. The ablation settles it.

Why not just use look-up-table networks or a smaller model on a commodity phone CPU? MuLUT's 100x energy advantage over FSRCNN is real, and it needs no new silicon. However, it is built around small receptive fields for SR and denoising, and the file reports no deblurring results for it. Its gains are measured against SR-LUT and FSRCNN, not against NAFNet-class quality. We compare on the energy-versus-PSNR frontier for SR, where both methods run. If MuLUT dominates there, we narrow our claim to deblurring and real-noise denoising, where large receptive fields matter.

Why not just run uniform int8 on a commercial edge NPU? Uniform int8 is what our design also uses, so this alternative is about the substrate, not the arithmetic. NPUs typically execute elementwise ops and normalization as separate memory-bound passes. We cannot measure a proprietary NPU in our flow, so the BrainTTA-style baseline stands in for a reduction-tree accelerator with a register file. This is a limitation we will state plainly.

Proposed Experiments

We train quantized NAFNet variants on GoPro (Deep Multi-scale CNN for Dynamic Scene Deblurring) and SIDD (A High-Quality Denoising Dataset for Smartphone Cameras). An SR variant is trained on DIV2K and tested on Set5 and Urban100. For each variant, we report PSNR after each hardware-motivated change: normalization folding, tile-local channel attention, and int8 with gate requantization.

On the hardware side, the metrics are energy per output pixel, DRAM bytes per pixel, the fraction of intermediate values carried by bypass, area, and 720p frame rate. Each is broken down into arithmetic, interconnect, register file, SRAM and instruction supply, which allows a direct comparison to BrainTTA's reported breakdown.

The decisive checks come in three steps. The first checks model feasibility: whether tile-local channel attention and int8 gating keep PSNR within a few tenths of a dB of the published NAFNet results. If not, the streaming premise fails before any hardware is built. The second checks whether direct bypass beats register-file routing under the identical fused schedule by a margin larger than the interconnect cost it adds. The third checks whether the interconnect pruned on one set of variants still compiles and runs held-out variants with modest slowdown. This last check separates a programmable co-processor from an expensively disguised fixed-function one.