Hardwired Distributed Processing Units for Inference: Neural Decisions at the Sensor that Free the Host

Raj MaliarXiv 2026 · arXiv

TL;DR

This paper introduces Hardwired Distributed Processing Units (HDPUs) that perform neural network inference directly at the sensor, completely bypassing the need for a host processor to handle raw video frames. By implementing a binarized CNN on a low-cost FPGA, the authors demonstrate real-time zebra-crossing detection that reduces data traffic to the host by over 38,000x while maintaining high accuracy and deterministic latency.

What Problem Does This Paper Address?

In traditional computer vision, cameras are "dumb" peripherals that stream every pixel to a host processor. This creates three major bottlenecks:

  1. Latency: The host must wait for an entire frame to arrive before it can even begin processing.
  2. Resource Drain: The host spends significant CPU cycles and energy fetching instructions and weights from memory to run the network.
  3. Data Congestion: Moving high-resolution frames (e.g., 614 KB per frame) clogs communication buses.

While previous work on HDPUs moved simple pre-processing (like edge detection) to the sensor, the final "decision" still rested with the host. This paper asks: Can the sensor itself make the neural decision?

Methods

The authors developed a dedicated hardware pipeline on a low-cost Xilinx Artix-7 FPGA. Unlike traditional accelerators, this HDPU has no processor, no operating system, and no full-resolution frame store.

Key technical components include:

  • Binarized CNN (BCNN): To fit within the limited logic of a small FPGA, the authors used a four-layer BCNN. This architecture replaces expensive multiply-accumulate operations with simple XNOR gates and population counts, drastically reducing the hardware footprint.
  • Fixed Datapath: The network weights and thresholds are hardwired into on-chip memory beside a fixed datapath. This ensures the unit decides on every frame in exactly 2,269,477 clock cycles (34.04 ms), regardless of the image content.
  • Hardware Auto-Contrast: Recognizing that real-world sensors often produce "flat" or low-contrast images, the authors integrated a hardware stage that calculates a 256-bin histogram and stretches the contrast in real-time before the pixels reach the neural network.
Fig. 1. Two ways to get a decision from a camera. (a) The host-centred routine moves every frame to the host, which fetches instructions and weights and runs the network. (b) The inference HDPU reduces the frame as it arrives and takes the decision at the sensor; the host receives 16 bytes and does no work.

Datasets

The model was trained and evaluated using:

  • Training Set: 11,281 images (7,552 with crossings) aggregated from 14 different sources.
  • T1 (Dashcam): 524 frames from the CDSet benchmark, representing real-world driving perspectives.
  • T2 (Street Photos): 115 high-quality street photographs from external sources to test generalization.
  • Verification: A bit-exact Python integer model was used to verify every stage of the RTL simulation and the final physical board implementation.

Key Insights and Ideas

  • The "Not-Sure" Band: Instead of a binary yes/no, the authors implemented a confidence margin. If the network's output score falls within a specific "band," the unit reports "not sure." This simple addition turned an average classifier into a highly reliable one, pushing accuracy to 95.7% for the images it chose to answer.
  • Sensor-Side Pre-processing is Non-Negotiable: The hardware contrast stage was vital. Without it, accuracy on dashcam frames plummeted to 47.7%; with it, accuracy recovered to 78.2%. This proves that for edge AI, the "signal conditioning" at the sensor is as important as the model itself.
Fig. 3. A frame captured from the board’s OV7670 (a), its 80 × 60 network input, which spans only a narrow range of gray levels (b), and the same input after the hardware auto-contrast stage (c).
  • Determinism as a Safety Feature: Because the HDPU is a hardwired pipeline, its timing is perfectly predictable. It never "lags" due to background OS tasks or complex image scenes, making it ideal for safety-critical applications like navigation aids for the visually impaired.

Key Results

  • Host Offload: The host processor is spared 8.13 ms of core time per frame. For comparison, a full-precision ResNet-18 would take longer than the entire frame period to run on the same host.
  • Massive Data Reduction: The data sent to the host drops from 614,400 bytes (raw frame) to just 16 bytes (the decision)—a 38,400x reduction.
  • Efficiency: The entire detection path uses only 1,814 Look-Up Tables (LUTs), a tiny fraction of even a low-end FPGA. The inference engine itself consumes an estimated 22 mW.
  • Performance: The unit processes 26.68 frames per second, matching the camera's native rate without skipping a single frame.

Limitations and Future Work

  • Power Measurement: The power figures are based on Vivado software estimates rather than physical measurements at the power supply.
  • Dataset Size: While the training set is robust, the T2 test set (115 images) is relatively small, leading to wider confidence intervals for those specific accuracy results.
  • Parallelism: The authors suggest that the next logical step is running multiple HDPUs (e.g., one for zebra crossings, one for Canny edges, one for Harris corners) in parallel on the same pixel stream to provide a comprehensive "smart sensor" output to the host.

Summary updated

Abstract

We show that a hardwired distributed processing unit (HDPU) can take a neural decision at the sensor and so remove the inference from the host. Built for zebra-crossing detection on a low-cost Artix-7 FPGA, the unit decides on every camera frame in a fixed 2,269,477 clock cycles (34.04 ms depends upon Fpga clk ), inside the 37.49-ms frame period, and on the board it returns the same scores and decisions as its bit-exact software model on all 639 test images. The whole detection path, from camera pixels to decision, takes 1,814 LUTs, with no processor, no operating system and no full-resolution frame store. When the unit answers instead of reporting "not sure", 95.7% of its answers on street photographs are correct, and a hardware contrast stage lifts accuracy on camera-like dashcam frames from 47.7% to 78.2%. The host is spared all of the inference: 8.13 ms of a core in every frame period, where a full-precision ResNet-18 would need more than a frame period even on 16 threads. It also receives 16 bytes per frame instead of 614,400, 38,400 times less. The unit is an 80x60-pixel pipeline feeding a four-layer binarized CNN with 93,408 one-bit weights, verified bit-exactly from a Python model through RTL simulation to the board. Earlier HDPUs did pre-processing (edges, corners, steering); this one shows that the decision itself can move to the sensor.
BibTeX
@misc{mali2026hardwired,
  author = {Raj Mali},
  title = {{Hardwired Distributed Processing Units for Inference: Neural Decisions at the Sensor that Free the Host}},
  year = {2026},
  eprint = {2611.06436},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06436},
}
Figure