Shaping Sparse AI Traffic on Shared Memory Buses

Edge chips increasingly run a neural network and a safety-critical control loop on the same silicon, sharing one memory bus. The standard protection is a traffic shaper: a hardware block that cuts long memory transfers from the AI engine into small chunks and caps how many bytes it may move per time window. These shapers were designed and evaluated against dense AI workloads, which stream large contiguous bursts. Pruned networks behave differently. In a model with unstructured sparsity, most weights are zero and the surviving ones sit at irregular positions, so inference proceeds by many small, scattered reads of index metadata and values. Our hypothesis is that a byte-budget shaper barely restrains this kind of interferer. The traffic is already "fragmented," so it passes the chunking rule. It moves few bytes, so it stays under the byte cap. Yet every one of its many short transactions still holds the bus and the memory banks, and those occupancy costs land on the critical task.

We propose to measure this effect, to build a shaper that charges each initiator for the bus time and bank time it occupies rather than for the bytes it moves, and to decide empirically whether the hardware change is needed at all or whether reformatting the sparse model into blocks gets the same result. The deliverable is a design rule for edge engineers. For a given bound on the critical task's latency, it should say which combination of sparsity format and shaping policy gives the most AI throughput.

Core Claim

The closest system is the 16nm mixed-criticality RISC-V chip of A Reliable, Time-Predictable Heterogeneous SoC for AI-Enhanced Mixed-Criticality Edge Applications. Its Traffic Shaper Unit fragments direct-memory-access (DMA) bursts and enforces per-initiator transfer budgets. Its cache partitioning keeps the AI workload from evicting the critical task's lines. Under an unregulated DMA flood the critical task slowed by 225×, and the shaper plus cache partitioning recovered 75% of isolated performance. That paper motivates this work, and its open issue is the one we target: the AI side is treated as opaque dense bursts, and the shaper's parameters are static and blind to sparsity.

Our claim is twofold. First, for a fixed shaper configuration, an unstructured-sparse interferer causes measurably more critical-task latency than a dense interferer of equal AI throughput. Second, a cost-weighted budget closes most of that gap. The budget charges initiator $i$ in each window

$$C_i = \sum_{k} \left(\alpha + \beta\, b_k\right) \le B_i,$$

where $k$ indexes the initiator's transactions in the window, $b_k$ is the bytes moved by transaction $k$, $\alpha$ is a fixed per-transaction charge reflecting arbitration and bank activation, $\beta$ is a per-byte charge, and $B_i$ is the window budget. Setting $\alpha = 0$ recovers today's byte budget, which makes that ablation exact.

The second claim is conditional. Its value disappears if blocked sparse formats already turn the scattered traffic into dense-looking bursts at little accuracy cost. Testing that possibility is part of the program, not something to argue around.

Research Program

The first step is characterization. Access to the published chip is not assumed. We will reimplement the shaper as the paper describes it (chunking plus per-initiator budgets), together with way-partitioned last-level cache, in a cycle-accurate RTL or FPGA-emulated RISC-V system. The system will have a vector AI engine and a small in-order core running the time-critical task, sharing an AXI bus to off-chip memory. We will calibrate this setup by reproducing the reported dense-DMA result: a large unregulated slowdown, and partial recovery once shaping is enabled. Our absolute numbers will differ from the 225× and 75% figures, but the qualitative shape must match before any sparse result is trusted.

We then replace the dense interferer with sparse inference kernels in several formats. Compressed sparse row (CSR) represents unstructured sparsity. A blocked variant follows the row-reordering idea of High Performance Unstructured SpMM Computation Using Tensor Cores, which clusters non-zeros into small dense blocks. We also include tile-wise sparsity from Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity, which prunes whole rows or columns inside matrix tiles, and 2:4 sparsity trained with SR-STE from Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch. These papers supply method, not motivation. All three measured only standalone speed and never the interference their memory traffic imposes on a co-running task.

The second step is the shaper. We implement the cost-weighted budget and fit $\alpha$ from measured bank-busy time per transaction on the emulated memory. We add one optional software hint: the sparse runtime writes a register declaring whether it is entering an index-heavy phase or a value-streaming phase, and the shaper switches budget tables accordingly. This is a deliberately narrow version of the software-to-hardware interface the chip paper lacks. A lightweight interface was needed regardless, as TEMPOS: QoS Management Middleware for Edge Cloud Computing FaaS in the Internet of Things shows. That work coordinated priority across middleware, network and OS scheduling, but explicitly left accelerators and memory subsystems as future work.

The third step is the design rule. For each critical-task latency bound, we report the Pareto set over format, sparsity level and shaper policy, with AI throughput and accuracy on one axis and critical-task tail latency on the other.

Baselines

The naive reference point is no shaping at all, with the critical task and sparse AI kernel co-running freely. It bounds how bad interference gets for each format. Next to it sits serialized execution: the AI workload runs only in time windows when the critical task is idle. This gives the best critical-task latency and the worst AI throughput.

The strongest existing method is the published shaper-plus-cache-partitioning design from A Reliable, Time-Predictable Heterogeneous SoC, reimplemented with its byte budget and chunking. Its parameters will be tuned by the same grid search, with the same number of configurations, that we use to tune the cost-weighted shaper. It is the strongest competitor because it is the only mechanism in the literature built for exactly this mixed-criticality bus. Its reported result on the dense interference benchmark is a recovery of 75% of isolated critical-task performance, against a 225× unregulated slowdown. We found no reported result on Lacuna for any shaper facing an unstructured-sparse interferer.

Beating this baseline means a strictly better frontier with an unstructured interferer. At equal AI inferences per second, the cost-weighted shaper must cut the critical task's 99.9th-percentile latency. Alternatively, at equal critical-task latency, it must deliver more AI throughput. In either case, accuracy must be unchanged, because the model itself is identical.

The format baselines change the model rather than the shaper. These are dense execution, 2:4, tile-wise and blocked-reordered sparsity, each under the tuned byte-budget shaper. Two ablations isolate the key ingredient. The first sets $\alpha = 0$ while keeping everything else, which shows whether the per-transaction charge is what matters. The second removes the phase hint, which shows whether static cost weighting suffices.

Why not just…?

Why not just enforce structured sparsity so the existing shaper sees ordinary bursts? This is the most serious threat. Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch reports 77.0% top-1 accuracy for 2:4 ResNet-50 against 77.3% dense. However, it acknowledges that the speedup requires sparse tensor-core hardware that edge vector units lack, so on our platform 2:4 may simply be dense traffic with wasted multiplies. Tile-wise sparsity runs on plain hardware but loses 1–3% accuracy at 75% sparsity Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity. At 90% sparsity, unstructured training retains more accuracy: RigL reaches 67.0% on MobileNet-v1 against 61.8% for conventional pruning Rigging the Lottery: Making All Tickets Winners. The Pareto experiment settles this directly. If tile-wise or 2:4 under the byte shaper matches the cost-weighted shaper's frontier at equal accuracy, we will recommend formats over hardware and report the shaper as unnecessary.

Why not just reorder the sparse matrix offline into dense blocks? For static inference weights, the preprocessing cost that High Performance Unstructured SpMM Computation Using Tensor Cores concedes is paid once, so this is a genuine competitor. It is included as a baseline format. The open questions are how much residual scatter remains after blocking on networks pruned by dynamic sparse training, and how much zero-padding it adds. Padding wastes bandwidth that the critical task also needs. The same paper notes that skewed non-zero distributions leave imbalance that reordering does not remove. If the blocked format erases the dense-versus-unstructured interference gap, the hardware contribution is moot for weights. The cost-weighted shaper would then matter only for input-dependent activation sparsity, which cannot be reordered offline Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity.

Why not just lower the byte budget until the critical task is safe? Tightening a byte budget throttles dense and sparse traffic alike. It should eventually protect the critical task, but at a large cost to AI throughput, because for scattered traffic bytes are a poor proxy for bus occupancy. The $\alpha = 0$ ablation, swept over budget sizes, is exactly this alternative. If its frontier coincides with the cost-weighted one, the per-transaction charge buys nothing.

Why not just serialize the two workloads? Sub-millisecond control deadlines, such as the roughly 500 µs strict-quality latency in TEMPOS, cannot wait behind multi-millisecond inference. Preemptive co-scheduling also schedules 10–12% more task sets than non-preemptive alternatives in Composite Resource Scheduling for Networked Control Systems. Serialization remains in the experiments as the latency floor.

Proposed Experiments

Interferer models are ResNet-18 and ResNet-50 at 80% and 90% sparsity and MobileNet-v1 at 90%. All are trained with RigL using its open recipe; RigL reports 74.6% top-1 for ResNet-50 at 80% sparsity Rigging the Lottery. Each is evaluated for ImageNet accuracy, then deployed as an inference loop on the emulated vector engine, with batch and resolution reduced to fit the memory. To stress the shaper beyond neural workloads, we add sparse matrix multiplication on SuiteSparse matrices with skewed non-zero distributions. The critical task is a pointer-chasing control kernel with a fixed deadline, matching the style of the published interference benchmark.

We measure the critical task's worst-case and 99.9th-percentile latency as a fraction of its isolated value, its deadline-miss rate, AI inferences per second, memory transactions per inference, and bank-busy cycles.

The first decisive check is the existence test. Under the tuned byte shaper, the unstructured interferer must cause higher critical-task tail latency than a dense interferer at equal throughput. If it does not, the premise fails and we stop at a negative characterization result, which is still useful to designers. The second check compares frontiers at equal accuracy: cost-weighted shaper with unstructured sparsity, against the byte shaper with tile-wise, 2:4 or blocked formats. The third check is the pair of ablations ($\alpha = 0$, no hint), which attributes any gain to its source. All three outcomes are informative, and the proposal commits in advance to reporting whichever format-versus-hardware answer the data give.