A large language model can be compressed by storing each weight with fewer bits. Recent methods can store weights at fractional average precisions such as 2.4 or 2.75 bits per weight, so a model's size can be set to match a device's memory exactly instead of jumping between 4, 3, and 2 bits. Every reported speedup for these formats was measured on a desktop RTX 4090. Edge chips behave differently. On an NVIDIA Jetson Orin, even ordinary 4-bit quantization makes text generation slower, because the chip spends more time unpacking weights than it saves by reading fewer bytes from memory. Bits per weight therefore predict memory but not speed on edge hardware. The format that loses the least accuracy at a given size may also be the slowest one to unpack on a small, power-limited processor.
We propose to choose, for each layer of a model, which quantizer to use from a mixed set of integer and fractional formats. The choice uses two quantities: the accuracy loss of that layer under each format, and the decode time of that layer under each format as measured on the target edge device at its current power setting. The memory budget is a hard constraint. The question we want to answer is simple: once speed is measured on the device rather than inferred from bit count, do fractional formats still win, and is the best allocation stable across power modes? If fractional formats lose, the method falls back to integer formats, and that negative result is useful in itself.
Core Claim
Our claim is that on embedded system-on-chips the cost of decoding a quantized layer depends on the format, the layer's shape, and the power mode. It does not follow bit count. As a result, allocations tuned for accuracy and bit count (or for desktop-GPU latency) will be clearly suboptimal in decode latency on a Jetson-class device at equal memory and accuracy, and the best allocation will differ between power modes.
The evidence that motivates this comes from Rethinking Small VLM Quantization. That paper profiled small vision-language models on Jetson Orin NX and AGX. INT4 backbones cut memory by 21.8% to 47.5% but slowed time-per-output-token by 10.4% to 56.3%, and BitsAndBytes kernel fragmentation made a vision encoder up to 4.66x slower. It tested only integer formats.
The fractional formats we would put on edge hardware come from three methods:
- Q-Palette builds trellis-coded quantizers, including a variant that mixes two bit-widths inside one layer, and selects schemes per layer with fusion-awareness.
- Double binary factorization writes each weight matrix as a product of two sign matrices, so inference uses additions instead of multiplications.
- LiftQuant projects weights onto a 1-bit lattice in a higher-dimensional space, which allows any bit-width.
All three validate speed only on an RTX 4090. That GPU's large caches and bandwidth hide exactly the unpacking costs that Shin et al. exposed on Jetson.
The closest prior allocation method is HAQ. HAQ searches integer bit-widths per layer using latency feedback from accelerator simulators such as BitFusion and BISMO. We differ in three ways: our candidate formats include fractional and factorized schemes, our latencies come from physical commercial edge silicon, and we treat the power mode as a variable rather than a constant.
Research Program
The work has three parts.
Porting and tuning decode kernels. We will run Q-Palette's quantizers, double binary factorization, and LiftQuant on Jetson AGX Orin and Orin NX. Both are Ampere-class GPUs sharing LPDDR memory with the CPU. We will also include native integer kernels from standard Jetson runtimes. To make the comparison fair, each format gets the same kernel-tuning budget, fixed in engineer-hours and autotuning trials. Whether the existing CUDA and Triton kernels run well on the Jetson architecture is an open engineering question. We will report the effort each port required, because a format that is hard to port is itself a finding about deployability.
Building per-layer latency tables. For each candidate format, each distinct layer shape in Llama-3.2-3B, Llama-3.1-8B, and Llama-3-70B, and each Jetson power mode (15W, 30W, 50W, MAX), we measure single-batch decode latency on the device. We measure under sustained load, because cross-platform measurements of vision-language-action models showed Orin throughput drops non-linearly as power modes are lowered.
Allocating formats per layer. Let $q_l$ be the format chosen for layer $l$. Let $D_l(q_l)$ be the accuracy loss it causes, estimated with Q-Palette's sensitivity procedure. Let $M_l(q_l)$ be the layer's memory and $T_l(q_l;p)$ its measured decode time at power mode $p$. We solve
$$\min_{\{q_l\}} \sum_l T_l(q_l;p) \quad \text{s.t.} \quad \sum_l D_l(q_l) \le \epsilon, \;\; \sum_l M_l(q_l) \le M,$$
where $\epsilon$ is the allowed accuracy loss and $M$ is the device's memory budget. We also solve the dual problem, minimizing loss under a latency budget. Both are knapsack-style integer programs that can be solved exactly at these layer counts.
The formulation assumes that layer latencies add up. This is a real assumption on a chip where fused kernels and shared caches create interactions between layers. We therefore test it directly by comparing summed table latency against end-to-end measured latency.
Baselines
Naive reference. The do-nothing point is uniform INT4 and INT3 weight-only quantization, run with the best native Jetson runtime kernels on the same models, prompts, and power modes. Uniform INT4 is what practitioners deploy, and Shin et al. report that it slows decode by 10.4% to 56.3% on Orin. Our method must beat it on decode latency at equal or lower memory and equal or better WikiText-2 perplexity.
Strongest integer competitor. This is layer-wise mixed integer precision in the style of Variable Layerwise Quantization. It assigns 2, 3, or 4 bits per layer to reach averages such as 2.85 bits while using only native integer kernels. No reported Jetson latency or WikiText-2 number was found on Lacuna for this method. We will run it at the same average bit targets and with the same calibration data as our method.
Strongest fractional competitors. These are Q-Palette's fusion-aware allocation, run unchanged with its RTX 4090-derived choices, and double binary factorization at about 2 bits. For the latter, the file reports WikiText-2 perplexity of 7.30 on Llama3-8B, against 7.84 for QuIP#. No reported WikiText-2 number for Q-Palette and no edge-device latency for any fractional format was found on Lacuna. We will therefore reproduce accuracy first and confirm we match the published figures before timing anything.
Ablation of the key ingredient. The ablation keeps our full candidate set and our solver but replaces the measured $T_l$ with a proxy. We use two proxies: bits times parameters, and the RTX 4090 latency table. Beating this ablation is what shows that on-device measurement, rather than the larger candidate set, is responsible for any gain.
What counts as success. We count the method as beating a baseline if it reduces sustained decode latency by more than the run-to-run variation we measure under thermal load, at matched memory and matched perplexity, in at least two of the four power modes.
Why not just…?
Why not just deploy uniform INT4 with tuned native kernels? It is simple and well supported, but the edge evidence works against it. On Orin it slowed decode rather than speeding it up (Shin et al.). At integer steps it also cannot use memory budgets well. LiftQuant notes that a 70B model at 3 bits exceeds 24GB while 2 bits wastes memory and loses accuracy (LiftQuant). The uniform-INT4 baseline will show whether these effects matter at our model sizes.
Why not just mix integer bit-widths across layers and skip fractional formats entirely? This is the most serious threat, and we do not argue around it. It reaches fractional averages using fast native kernels. If decoding within-layer fractional formats is expensive on Orin, this approach may simply win. Our allocator includes the integer formats in its candidate set, so it can never do worse than this baseline. The experiment that settles the question is how often the solver picks a fractional format when latency is measured. If it rarely does, we will report that within-layer fractional formats are not worth their decode cost on this class of hardware.
Why not just use an analytic roofline model instead of measuring? A roofline model estimates latency from memory bandwidth and arithmetic throughput. If it ranks formats correctly per layer and per power mode, the measurement campaign is unnecessary. We will fit one and report its rank correlation with measured $T_l$. If the correlation is high and the resulting allocations match, we will recommend the cheaper roofline approach.
Why not just use BitStack for memory flexibility? BitStack resizes a model at runtime by stacking 1-bit residual blocks. It answers a different question: adapting to memory that changes over time, not choosing the fastest format for a fixed budget. We include it as a format option at matched sizes, measuring both its accuracy and its decode latency on Orin.
Proposed Experiments and Decisive Checks
We evaluate WikiText-2 perplexity and zero-shot accuracy on PIQA, ARC-Easy, ARC-Challenge, HellaSwag, and WinoGrande. On the device we measure sustained time-per-output-token, peak memory, and energy per token from the board's power rails. The test matrix covers Llama-3.2-3B and Llama-3.1-8B on Orin NX and AGX, and Llama-3-70B on AGX near a 24GB budget, so results can be compared with LiftQuant's 4090 setting of 31.3 tokens per second.
Three checks determine whether the claim holds:
- Proxy-induced latency gap. We compare the latency of proxy-chosen allocations with that of measurement-chosen allocations at matched accuracy. A gap no larger than measurement noise falsifies the core claim.
- Power-mode dependence. We test whether the allocation that is best at MAX mode is measurably beaten at 15W by an allocation re-solved for 15W. If it is not, a single offline allocation is enough.
- Latency additivity. We test whether per-layer table latencies sum to the end-to-end decode time within a small error. If they do not, we replace the additive solver with search guided by end-to-end measurement.
A small vision-language model from Shin et al., scored on MME, serves as a secondary test. It checks whether the vision encoder, where that paper found its kernel-fragmentation slowdown, should be kept out of fractional formats entirely.
References
- Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment
- Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment
- Addition is almost all you need: Compressing large language models with double binary factorization
- LiftQuant: Continuous Bit-Width LLM via Dimensional Lifting and Projection
- HAQ: Hardware-Aware Automated Quantization
- Cross-Platform Scaling of Vision-Language-Action Models from Edge to Cloud GPUs
- Variable Layerwise Quantization: A Simple and Effective Approach to Quantize LLMs
- BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments