Edge servers that run AI models for 5G network slices, meaning isolated virtual networks each promised its own latency and reliability, increasingly share one neural accelerator chip among many tenants. The newest accelerator schedulers lend idle compute units from one tenant to another to raise utilization. Separately, hardware designers have found that once weights and activations are compressed to 8 bits or fewer, the expensive part of the chip is no longer the multiplier. It is the 32-bit register that sums the products, called the accumulator. Chips with narrow accumulators of 12 or 16 bits are much cheaper, but a model is only safe on them if its weights were trained or quantized to keep every running sum inside that register. These two trends collide. A scheduler that lends a narrow-accumulator unit to a tenant whose model was prepared for a wider one will silently corrupt that tenant's outputs through integer overflow.
We propose to attach to each quantized model a small, provable "overflow certificate": a per-layer table, computed from the weights alone, that states which accumulator widths the layer can run on and at which tile sizes. The tile size is the number of products summed before the partial result is handed to a wider second-stage register. A slice-aware scheduler can then dispatch work onto accelerators whose units have different accumulator widths. It re-tiles a layer at dispatch time to fit whatever unit is free, with a mathematical guarantee of zero overflow and no re-quantization. The claim is that this makes accumulator width a resource the scheduler can safely allocate. At equal silicon area, a chip mixing narrow and wide units should serve more slices within their latency targets than a chip where every unit uses one width.
Why this is open, and what already exists
Three papers supply the method. A2Q showed that the accumulator width a dot product needs is bounded by the $\ell_1$-norm of its weight vector, not by the data type's worst case. By constraining that norm during training, it reached 12- and 8-bit accumulators with a 2.3x cut in FPGA lookup tables. A2Q+ doubled the usable budget by forcing weights to sum to zero, keeping 95% of ResNet50's ImageNet accuracy at 12 bits. AXE brought the same bound to post-training quantization of large language models. It also introduced multi-stage accumulation, in which long dot products are cut into tiles that are summed narrowly and then combined in a wider register. All three target one fixed datapath per model. None asks what happens when the datapath a layer lands on is chosen at run time by a scheduler.
Two papers supply the sharing regime and motivate the gap. Neu10 splits accelerator instructions into small tensor operations so that idle matrix units can be lent between collocated tenants, gaining up to 1.4x throughput. vNPU partitions interconnected accelerator cores among tenants and reports 1.92x over static partitioning on Transformers. Both treat every matrix unit as arithmetically identical. Network slicing orchestrators go further in abstraction. Sl-EDGE models edge compute as CPU share. Slicing for AI trades accuracy through training epochs and data volume on generic virtual CPUs. LLM-Slice treats the inference server as a fixed delay block behind a radio scheduler. Our proposal differs from all of these in one specific way. It lets arithmetic width vary per compute unit and makes the scheduler prove, before each dispatch, that the width is safe for the layer.
One paper constrains the design. A2Q reports that simple overflow handling, by wrapping around or by clamping the sum, "fails miserably" once accumulators shrink, and AXE reports that naive width reduction loses up to 15% in WikiText2 perplexity. So the scheduler cannot simply let overflows happen and hope that clamping hides them. Safety has to be established before execution.
Core Claim
For a layer with integer weights $w$ split into tiles of size $T$, let $B_T(w)$ be the largest $\ell_1$-norm over its tiles. Let $x_{\max}$ be the largest activation magnitude, which the activation bit-width fixes. Following the A2Q bound (up to its rounding conventions), the inner accumulator needs roughly
$$P_{\text{inner}}(T) \approx \lceil \log_2 (B_T(w)\, x_{\max} + 1) \rceil + 1$$
bits, where the final $+1$ is the sign bit. The outer accumulator, which sums $\lceil K/T \rceil$ tile results for a layer of width $K$, needs about $\log_2 \lceil K/T \rceil$ more bits than that. Two facts make a run-time certificate practical. First, $B_T(w)$ can only fall when $T$ shrinks, and each doubling of $T$ at most doubles it, costing at most one extra bit. The certificate is therefore a short, monotone table over a few candidate tile sizes, computed once from the quantized weights. Second, checking whether a free unit with inner width $P$ and outer width $Q$ can run a layer is a table lookup, which adds essentially no time to a dispatch decision. The research bet is that one set of weights, quantized once toward the narrowest target, remains accurate while being certified for a whole family of (width, tile) datapaths. That family is what the scheduler exploits.
Research Program
The work has three parts. The first is multi-target quantization. We extend AXE's $\ell_1$-ball projection and the A2Q+ zero-centering so that each tenant's model is quantized against the tightest target in the family, for instance 12-bit inner accumulation at small tiles. We then measure its accuracy on every wider (width, tile) point it is certified for, against a model quantized natively for that point. We expect to find that the cost of portability is small for wide layers and large for narrow ones. Depthwise convolutions with $K=9$ are the known failure case of A2Q+, where zero-centering removes too much freedom. For such layers, the certificate will simply pin them to wide units. How much of a real model ends up pinned is itself a measurement worth reporting.
The second part is a certificate-checking dispatcher. We add it to a Neu10-style cycle-level accelerator simulator whose matrix units are given heterogeneous accumulator widths. Each unit's area comes from a gate-count model calibrated to the roughly 63% fused-multiply-add gate saving that Blumenfeld et al. report for 12-bit versus 32-bit accumulators. The dispatcher lends idle units across tenants as Neu10 does. It is allowed to re-tile a layer to fit a narrower lender, but only when the certificate permits. The simulator will count every overflow bit-exactly, so that the guarantee is audited rather than assumed.
The third part is a slice-level controller. Each network slice carries a latency target and an accuracy floor. The controller decides which mix of narrow and wide units each slice's tenant may borrow. We will adapt the online bandit controller of Slicing for AI so that its actions include accumulator-width classes rather than only virtual CPU shares.
Baselines
The do-nothing reference is a chip whose matrix units all have 32-bit accumulators, running tenants quantized to 8-bit weights and activations, scheduled by Neu10-style harvesting. It will never overflow, and it defines the accuracy ceiling and the area cost we aim to undercut. Following Neu10, it is the strongest published multi-tenant accelerator scheduler in the file, with reported gains of up to 1.4x inference throughput and 4.6x lower 95th-percentile tail latency over unshared execution. We reimplement its harvesting policy rather than compare against its reported numbers, because their simulator and our area model differ.
The strongest competitor is a uniform narrow chip: every unit uses AXE's 16-bit multi-stage datapath, every tenant is quantized with AXE for that single target, and the same harvesting scheduler runs on it. AXE reports that Llama3 8B on this datapath keeps up to 98% of FP16 WikiText2 perplexity and 99% of 32-bit zero-shot baselines. This is the configuration most likely to make our heterogeneity unnecessary.
All systems get the same die area, the same tenant mix and arrival traces, and the same post-training quantization calibration set and tuning budget per tenant. To isolate the key ingredient, we add two ablations on the heterogeneous chip. The first is width-oblivious harvesting, which lends any unit to any tenant; it measures how often overflow actually occurs and how much accuracy it destroys. The second is certificate-only dispatch without re-tiling, which uses a unit only if the layer's native tile already fits.
Beating the uniform 16-bit chip means one of two outcomes at equal area and a matched accuracy floor per tenant (within one point of that tenant's result on the uniform chip). Either the heterogeneous chip admits at least 20% more slices that meet their 95th-percentile latency targets, or it serves the same slices with a comparable reduction in accumulator area. In both cases the audited overflow count must be exactly zero.
Why not just…?
Why not just keep 32-bit accumulators everywhere? This is the zero-risk option, and it is the do-nothing baseline. The evidence against it is area and power. The 12-bit accumulators of Blumenfeld et al. cut gate count by about 63%, with ResNet-50 top-1 at 76.22% versus 76.80%, and A2Q reports a 2.3x reduction in lookup tables. At a fixed die area, that saving buys more matrix units, and our iso-area experiment measures whether the extra units translate into admitted slices.
Why not just make every unit 16-bit with AXE and skip heterogeneity? This is the real threat. If AXE's single 16-bit tiled datapath already keeps near-full accuracy for all tenants, a mixed chip may add scheduling complexity for little gain. The argument for heterogeneity is twofold. Some tenants, such as the CIFAR-10 and ImageNet ResNets under A2Q+, tolerate 11 to 12 bits and would waste area on 16-bit units. Other layers, such as depthwise convolutions, may need wider units. Whether this matters in practice depends on the tenant mix, so we will report the full curve of slice admission against the fraction of narrow units. If the uniform 16-bit chip is within a few percent across realistic mixes, we will report that as the finding and narrow the claim to certificates for safe harvesting on uniform narrow chips.
Why not just profile each tenant on each width offline and dispatch from the profiling table? Profiling is simpler than certificates and would capture accuracy well on the calibration data. It cannot rule out overflow on inputs it never saw, and A2Q's wrap and clip results together with the corrupted feature maps documented by Xie et al. show how costly a rare overflow is. We test this directly by building a profiling-based dispatcher and counting overflows and accuracy loss on held-out and shifted inputs. Shifted inputs include ImageNet images outside the calibration set and long WikiText2 contexts. The certificate predicts zero overflows by construction.
Why not just store several copies of each model, one quantized per width? Copies avoid the portability cost but multiply weight memory, which is scarce on edge servers, and each copy is still tied to one tile size. We measure the accuracy gap between one portable model and per-width copies. If the gap exceeds one point on a tenant, per-width copies are the honest recommendation for that tenant.
Proposed Experiments
The tenant set covers the benchmarks in the file. Llama3 8B is evaluated on WikiText2 perplexity and the zero-shot suite of ARC-Challenge, HellaSwag, PIQA and Winogrande, following AXE. ResNet50 and MobileNetV1 run on ImageNet-1K. ResNet18 runs on CIFAR-10, where A2Q+ reports about 90% at 11 bits versus about 81% for A2Q. ESPCN runs on BSD300 PSNR. The 11-layer model on the mHealth dataset from Slicing for AI stands in for a small telemetry slice. Arrival patterns for the language-model slice will follow the bursty response behavior described by LLM-Slice. That paper releases no trace we can cite, so we synthesize bursts and state that assumption openly.
The first decisive check is the portability cost. For each tenant, we compare accuracy under one weight set certified across the (width, tile) family against native AXE or A2Q+ quantization at each point. We also report the fraction of layers pinned to wide units.
The second decisive check is the audited safety claim. The certificate-checking dispatcher must record zero overflows over all benchmarks and shifted inputs. Width-oblivious harvesting and profiling-based dispatch are measured on the same inputs for comparison.
The third decisive check is the iso-area systems result. We report slices admitted within their 95th-percentile latency target, SLA violation rate and per-tenant accuracy for four chips: the uniform 32-bit chip, the uniform 16-bit AXE chip, and the heterogeneous chip with and without re-tiling. The comparison is swept over the fraction of narrow units.
We also report the scheduler's decision time per dispatch, to confirm that certificate lookup stays negligible next to the tensor operation it schedules. The accumulator-area savings rest on the gate-count and lookup-table figures above. No reported result was found on Lacuna for slice-level SLA metrics on any accelerator with narrow accumulators, so all systems-level numbers will be new measurements.
References
- A2Q: Accumulator-Aware Quantization with Guaranteed Overflow Avoidance
- A2Q+: Improving Accumulator-Aware Weight Quantization
- Accumulator-Aware Post-Training Quantization for Large Language Models
- Hardware-Assisted Virtualization of Neural Processing Units for Cloud Platforms
- Topology-Aware Virtualization over Inter-Core Connected Neural Processing Units
- Sl-edge: network slicing at the edge
- Slicing for AI: An Online Learning Framework for Network Slicing Supporting AI Services
- LLM-Slice: Dedicated Wireless Network Slicing for Large Language Models
- Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators
- Overflow Aware Quantization: Accelerating Neural Network Inference by Low-bit Multiply-Accumulate Operations