Edge accelerators fail abruptly when a model does not fit in on-chip memory. A Google Edge TPU has 8 MB of fast on-chip memory (SRAM). A network that fits runs an inference in about 2 ms. Once a network crosses that line, it must fetch data from external memory, and its speed and energy efficiency collapse toward those of an embedded CPU, which needs close to 100 ms for the same job (Exploring edge TPU for network intrusion detection in IoT). The research on recovering accuracy after compression treats the problem as smooth instead: pick 50% sparsity or 4-bit weights, then minimize the error. That framing misses two facts about real deployment. First, any byte saved below the hardware limit buys nothing extra. Second, the limit covers more than the weights. It also covers the peak memory needed for intermediate results (activations), and pruning weights does not automatically shrink that peak (SpArSe; TinyTL).
This note proposes compressing a pretrained network so that it fits the exact byte budget of a target chip, for weights and peak activations together, and then repairing the accuracy damage with a closed-form linear correction that adds no memory at all. The method removes whole channels, because removing a channel shrinks both the weight tensor and the activation map. It decides where to cut based on which of the two memory constraints is actually binding, rather than spreading cuts evenly. It relies on the fact that the repair step from GRAIL is cheap enough to run inside the search for every candidate cut. The goal is to get close to the accuracy of training-heavy, chip-aware compression in minutes of calibration instead of GPU-days.
Core Claim
GRAIL is a post-hoc repair method. It records how often the original model's channels fire together on 128 unlabeled samples, summarized as a Gram matrix. It then solves a ridge regression that predicts the pruned channels from the surviving ones, and folds the resulting map into the next layer's weights. On the ResNet-18 setting it reports, this restores accuracy from 17.6% to 84.8% at 65% channel removal, with no backpropagation and no added latency.
We claim that a budget-driven placement of the cuts beats uniform per-layer ratios at the same byte budget. The rule is: place cuts on the layers whose removed channels free bytes against whichever constraint binds, and rank candidates by error measured after GRAIL's repair. The reason is structural. Weight memory is a sum over layers, but peak activation memory is a maximum over the execution schedule. When the activation constraint binds, channels removed from layers outside the peak region cost accuracy and free no useful memory.
Written as an optimization problem, we choose per-layer channel keep counts $k_l$ to minimize the post-repair reconstruction error $E(k)$ on calibration data, subject to
$$ \sum_{l} b\, P_l(k_{l-1},k_l) \;+\; \max_{t} A_t(k) \;\le\; M, $$
where:
- $P_l$ is the parameter count of layer $l$,
- $b$ is the bytes per parameter after 8-bit quantization,
- $A_t(k)$ is the bytes of activations alive at schedule step $t$ under the deployment compiler's buffer plan,
- $M$ is the chip's on-chip capacity.
What is new is not the repair step itself, which we reuse. It is treating the repair as an inner loop inside a hard, two-part memory constraint, and validating the result on hardware that has a real memory cliff.
Research Program
Profiling memory. We first build a memory profiler for each target that returns $A_t(k)$ under the actual schedule. Three targets are planned:
- The Edge TPU compiler's on-chip allocation report.
- The buffer plan of the uTensor embedded runtime used in SpArSe, on STM32-class microcontrollers.
- The Block RAM (BRAM) usage reported by hls4ml for field-programmable gate arrays (FPGAs).
For residual networks, the profiler also applies layer-centric tile fusion (Memory-Efficient Feature Merging for Residual Connections with Layer-Centric Tile Fusion). We do this so that we only spend accuracy on activation peaks that scheduling alone cannot remove.
Allocating cuts. The allocator is greedy and works in byte units. At each step it considers removing a block of channels from each layer and computes the GRAIL-repaired error for that layer pair in closed form. It credits each candidate only with bytes freed against the currently binding constraint. It then commits the candidate with the lowest added error per useful byte. The loop stops the moment the model fits.
Spending leftover memory. If the model fits with room to spare, the allocator returns channels to layers in reverse order of error-per-byte. As an optional second step, it spends the remaining bytes on a small eigenspace low-rank correction in the style of EoRA. Each correction is charged its true weight cost and its intermediate-activation cost, so the comparison between compressing more and correcting more is made in the same currency.
Choosing structured cuts. We use structured pruning on purpose. LLMCBench reports that sparsity which merely zeroes weights does not reduce memory footprint on real engines, while physically removing structure does. Deep Compression needed a dedicated accelerator (EIE) to benefit from irregular sparsity. Dense, narrower layers map directly onto Edge TPU systolic arrays and onto FPGA BRAM.
Models and budgets. We study three settings, each at two or three budgets per device:
- ResNet-18 and DeiT-Small on ImageNet-1k, deployed on the Edge TPU at 8 MB.
- CNNs on CIFAR-10 and the SpArSe microcontroller suite, deployed on STM32-class parts with budgets below 1 MB.
- The ToN-IoT intrusion-detection models from the Edge TPU study, which already live near the cliff.
We avoid budgets below roughly 25 KiB for CNNs. The Edge TPU study's "small model paradox" shows that at that size the accelerator is slower than the CPU anyway, so fitting it buys nothing.
Baselines
Naive reference. The do-nothing baseline is uniform 8-bit post-training quantization with no pruning, the setting profiled in the Edge TPU study. There, ToN-IoT models reach an F1 score of about 0.98, the TPU takes about 2 ms near the 8000 KiB mark versus nearly 100 ms on the CPU, and accuracy degrades for models above 6000 KiB. For ImageNet models that exceed 8 MB at 8 bits, this baseline is run off-chip, and its measured latency and energy show the size of the cliff we are trying to avoid.
Most direct competitor. The real threat is the same structured pruning plus GRAIL repair, but with one uniform keep ratio for every layer, raised until the model fits. It uses the identical calibration set (128 unlabeled samples), the same 8-bit quantization, the same tile-fused schedule, and the same memory profiler. The only difference is how cuts are placed, so this baseline isolates our key ingredient. The same 128 samples are used for every method in this comparison. Our method beats it only if it achieves higher top-1 accuracy at the same feasible budget on at least two of the three hardware targets. Its main advantage should appear in activation-bound regimes. A second ablation keeps our allocator but ranks candidates by error before repair. This tests whether running the repair inside the search matters.
Strongest training-based competitor. The strongest existing chip-aware method is DeepCompress-ViT. It trains a learned encoder-decoder over the weights with distillation, then reorders the matrix products at inference. It reports 14.9× compression of DeiT-Small (84.1 MB to 5.64 MB) with less than 1% ImageNet accuracy loss, fitting an 8 MB budget, and up to 1475× energy reduction and 68.4× speedup for DeiT-Base on simulated 8 MB hardware. We compare at matched on-chip budget and report accuracy together with GPU-hours spent on compression. Two outcomes would count as a win:
- Accuracy within about one point of DeepCompress-ViT at under 1% of its compute.
- Equal accuracy once activation peaks are counted. DeepCompress-ViT budgets weights only and leaves patch embeddings and LayerNorm uncompressed.
We do not expect to beat it outright on accuracy.
Search-based competitor. On microcontrollers the comparison is SpArSe. It runs multi-objective Bayesian architecture search over error, Flash, and peak RAM, and reports models up to 7.4× smaller than the Bonsai and ProtoNN baselines on CIFAR10-binary within 2 KB RAM and 32 KB Flash. We will reuse its benchmarks and its uTensor deployment path. No per-dataset accuracies for CUReT, Chars4k, or USPS-binary were found on Lacuna, so we will rerun SpArSe's released configuration where possible.
Why not just…?
Why not just use 4-bit or lower post-training quantization until the model fits? Bit-width cuts do shrink weights. However, 8-bit is the Edge TPU's native format, and the Edge TPU study shows that 8-bit models still cross the 8 MB limit as they scale. Below 4 bits, accuracy collapses: EoRA reports gains of 10.84% on ARC-Challenge and 11.45% on GSM8K when repairing 3-bit LLaMA3-8B, which measures how much uncorrected quantization had lost. DeepCompress-ViT reports that standard methods lose nearly 20% at the ratios needed for DeiT-Small. Quantization also does not shrink activation peaks unless activations are quantized too. We will still include a 4-bit-weight variant wherever the toolchain allows it, as an extra baseline.
Why not just prune and then fine-tune with LoRA? LoRA, which adds small trainable low-rank matrices alongside frozen weights, needs labels, gradients, and tuning. RestoreLCC reports that such parameter-efficient methods often failed to converge at 60% sparsity on LLaMA-7B. RestoreLCC itself beat the best of them by 2.01% at 50% sparsity. Unless the adapters are merged back into the weights, they also add inference-time activation memory, which our budget charges. Still, fine-tuning after allocation can only help. We will report a short LoRA-then-merge stage on top of both our allocator and the uniform baseline. This tests whether a better allocation still matters once training is allowed.
Why not just stream weights from external memory instead of insisting they fit? Streaming pays the off-chip energy cost: about 640 pJ per DRAM access versus 5 pJ per SRAM access, according to Deep Compression. The DeepCompress-ViT edge results come precisely from avoiding that fetch. Tools like Auto WS make streaming practical, but residual feature maps still spill to external memory. Our Edge TPU measurements just above and just below the limit will quantify this trade directly on real silicon.
Why not just run tiny-hardware architecture search? SpArSe needs multiple GPU-days and has only been shown on small CNNs. Our allocator starts from a pretrained network and evaluates each candidate with a closed-form solve. On the microcontroller suite this is a genuine contest, and SpArSe may win on accuracy. Our claim there is a better trade between accuracy and search cost, and we will report both numbers.
Decisive Experiments
Checking the cliff and the profiler. For each Edge TPU model, we produce variants at 95%, 100%, and 105% of the 8 MB on-chip capacity. We measure latency and energy to confirm that the profiler's joint constraint predicts which side of the cliff each variant lands on. If predictions miss, the whole budget formulation is wrong. This is the first check.
Testing whether allocation matters. We sweep the budget $M$ and plot accuracy against $M$ for our allocator versus uniform keep ratios. Both use GRAIL repair, the same calibration set, and the same schedule. We expect the largest gaps where activations bind: early, high-resolution ResNet layers and ViT token-activation peaks. If the gap stays below half a point everywhere, the proposal reduces to "uniform pruning plus GRAIL fits on-chip," and we will report that as the finding.
Locating where closed-form repair fails. GRAIL reports that repair breaks down above 90% pruning. We will record the smallest budget per model at which repaired accuracy stays within five points of dense accuracy. That threshold tells practitioners which chip a given model can live on without retraining.
Matched comparison with training. On DeiT-Small at 8 MB, we report accuracy, peak activation bytes, and compression GPU-hours against DeepCompress-ViT. This includes our variant that spends leftover memory on low-rank corrections. The comparison settles whether training-free, budget-aware compression is a credible substitute.
References
- Exploring edge TPU for network intrusion detection in IoT
- SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
- TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning
- GRAIL: Post-hoc Compensation by Linear Reconstruction for Compressed Networks
- hls4ml: A Flexible, Open-Source Platform for Deep Learning Acceleration on Reconfigurable Hardware
- Memory-Efficient Feature Merging for Residual Connections with Layer-Centric Tile Fusion
- EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation
- LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment
- Deep Compression: Compressing Deep Neural Networks with Pruning
- EIE: Efficient Inference Engine on Compressed Deep Neural Network
- DeepCompress-ViT: Rethinking Model Compression to Enhance Efficiency of Vision Transformers at the Edge
- Restoring Pruned Large Language Models via Lost Component Compensation
- Auto WS: Automate Weights Streaming in Layer-Wise Pipelined DNN Accelerators