Quantizing the Router: Compressing Coarse-to-Fine Vision Transformers Without Breaking Where They Look

Two of the most effective ways to make vision transformers cheap are usually studied apart. A vision transformer is an image model that cuts the picture into patches and lets every patch attend to every other. The first trick is coarse-to-fine processing. The model looks at a cheap low-resolution view, stops early if it is confident, and otherwise "zooms in" at full resolution only on the region its attention says matters. The second trick is quantization, which stores and computes with low-bit integers instead of 16-bit floats. Compounding the two looks free, but it is not. The coarse stage makes its decisions to stop or zoom, and where to zoom, from its own confidence and attention maps. Quantization is known to scramble exactly those signals.

This note proposes to measure that failure and fix it. Compression error in a dynamic model leaks out through two channels. One is the familiar direct loss of accuracy. The other is misrouting: false early exits, unnecessary zooms, and zoom windows placed on the wrong region. Misrouting costs accuracy and also quietly takes back the compute savings. The concrete contribution is a post-training quantization procedure whose calibration objective explicitly preserves the routing decisions of the full-precision model. It is paired with a stage-level precision split that standard integer kernels can actually execute. Together these should give a better accuracy-per-compute frontier than quantizing either a static or a dynamic model naively.

Core Claim

Coarse-to-fine methods are evaluated only at floating-point precision. LF-ViT runs a 112×112 view first. When confidence falls below a threshold, it extracts a square high-resolution neighborhood chosen by class attention. This cuts DeiT-S from 4.6G to 1.7G FLOPs at 79.8% top-1. CF-ViT re-splits only the patches ranked highest by a moving average of class attention. Both motivate this proposal, because both make the costly decisions from attention statistics.

Separately, Post-Training Quantization for Vision Transformer shows that quantization flips the relative ranking of attention scores and introduces a ranking loss to counter it. That finding constrains our design: it predicts that the coarse stage's choice of where to zoom is fragile. Our claim is that the right thing to preserve when compressing a dynamic model is its decisions, not its layer outputs. Those decisions are the exit choice and the top-K zoom set.

A simple cost model shows why the two channels couple. Let $C_c$ and $C_f$ be the operation counts of the coarse and fine stages, $b_c$ and $b_f$ their bit-widths, and $p$ the fraction of images escalated to the fine stage. The expected bit-operation cost per image is then

$$ \mathbb{E}[\text{cost}] = C_c\, b_c + p\, C_f\, b_f . $$

Quantizing the coarse stage lowers $b_c$. If the low-bit coarse stage becomes less confident, $p$ rises and the larger $C_f b_f$ term grows, so the net saving can be small or even negative. If the coarse stage instead becomes overconfident, $p$ falls but accuracy drops through false exits. Neither effect appears in static-model quantization studies.

The closest existing work is Patch-wise Mixed-Precision Quantization of Vision Transformer (PMQ). PMQ assigns higher bits to attention-important patches within a fixed 224×224 input. It never changes the token count, never makes exit or zoom decisions, and relies on per-patch mixed precision that is hard to speed up on real hardware. We differ on all three counts. The resolution decision is the object being protected. Precision is split per stage rather than per patch. And we report wall-clock latency, not only bit counts.

Research Program

The first phase is diagnostic. We take released LF-ViT and CF-ViT models on DeiT-S and LV-ViT backbones and apply standard per-layer post-training quantization at 8, 6 and 4 bits. Each stage is quantized separately and then jointly. For every setting we record three routing statistics against the full-precision model. The first is exit agreement: the fraction of images whose stop-or-zoom decision matches. The second is zoom-region overlap: the intersection-over-union between the quantized and full-precision zoom windows (or, for CF-ViT, re-split patch sets). The third is the change in escalation rate $p$.

The diagnostic hypothesis is that 4-bit coarse stages show low zoom-region overlap well before final accuracy collapses, and that this hidden misrouting accounts for a large share of the accuracy lost. To test that share causally, we keep the quantized model but substitute the full-precision routing decisions ("oracle routing"). The accuracy gap closed by this substitution is the share attributable to misrouting.

The second phase is the method, routing-preserving calibration. During calibration on the usual small unlabeled subset, the quantization scales and rounding are fit to minimize

$$ \mathcal{L} = \lVert z_q - z_{fp} \rVert^2 + \alpha\, \mathcal{L}_{\text{topK}}(a_q, a_{fp}) + \gamma\, \mathrm{KL}\!\left(s_{fp} \,\Vert\, s_q\right). $$

The terms are defined as follows. The quantities $z_q$ and $z_{fp}$ are the quantized and full-precision final features. $a_q$ and $a_{fp}$ are the coarse-stage class-attention scores that select the zoom region. $\mathcal{L}_{\text{topK}}$ is a margin loss that penalizes any patch in the full-precision top-K set being ranked below any patch outside it. $s_q$ and $s_{fp}$ are the coarse-stage softmax outputs that drive the exit threshold. $\alpha$ and $\gamma$ weight the routing terms.

The top-K term departs deliberately from the all-pairs ranking loss of Liu et al.. The router only needs the set boundary to stay put, not the full ordering. That leaves calibration capacity free for the rest of the network.

The precision split is searched over a small grid of stage-level pairs $(b_c, b_f)$, for example 4/8, 6/8, 4/6 and 8/4. For each pair we keep the configuration on the accuracy-versus-measured-latency frontier. Because precision is uniform within a stage, both stages run on ordinary INT8 or INT4 kernels. This avoids the hardware mismatch that limits per-token mixed precision. If routing-preserving calibration leaves a residual gap at 4 bits, we will test a zero-overhead fallback borrowed from RestoreLCC. RestoreLCC folds a learned correction direction for lost activations into attention-head biases. We would apply that correction only to the coarse stage's class-attention heads. This is an optional add-on, not the core claim.

The third phase tests whether the effect grows when fine detail matters more. Here we fine-tune the dynamic models at higher input resolution on fine-grained datasets. Convolutional Low-Resolution Fine-Grained Classification shows that downsampling hurts sharply on these datasets: AlexNet reaches 50.4% versus RACNN's 63.8% on Stanford Cars, and 51.3% versus 58.1% on CUB-200. Zoom placement should therefore matter more on them than on ImageNet.

Baselines

The naive reference point is the full-precision coarse-to-fine model itself. For LF-ViT on DeiT-S this is 79.8% top-1 at 1.7G FLOPs on ImageNet-1K, as LF-ViT reports. It also includes the full-precision static DeiT-S at 4.6G FLOPs. Every quantized result is reported as a point on accuracy versus bit-operations and versus measured latency against these two anchors. The "do nothing to the router" baseline applies standard per-layer post-training quantization to the dynamic model with the original exit threshold. It isolates how much damage arrives through routing.

The strongest static competitor is a static ViT quantized with the best method in the file. Post-Training Quantization for Vision Transformer reports 81.29% top-1 for 8-bit DeiT-B, against 81.8% at full precision. PMQ reports 76.68% for DeiT-S at about 6 bits on average. These are the right opponents because the alternative to combining the two tricks is to use only quantization. To keep the comparison fair, all methods use the same backbone family, the same calibration images and size, and the same search budget over hyperparameters. Static models are compared at matched bit-operations and matched measured latency, not at matched nominal bit-width.

The strongest dynamic competitor is no published method, because no reported result was found on Lacuna for quantized LF-ViT or CF-ViT. We therefore construct it. It is the ranking-aware PTQ of Liu et al. applied directly to the dynamic model, with the exit threshold re-tuned on held-out data so the escalation rate matches the full-precision $p$.

Beating the baselines means two things at 4/8 stage precision. Accuracy must sit within 0.5 points of the full-precision LF-ViT. Measured latency must be lower than both static quantized DeiT-S and the re-tuned dynamic baseline at equal or better accuracy. Two ablations isolate the key ingredient. One removes the top-K term while keeping the confidence term. The other removes the confidence term while keeping the top-K term. A further check swaps in the all-pairs ranking loss in place of the top-K term at equal calibration cost.

Why not just…?

Why not just re-tune the exit threshold after quantizing? This is the most serious threat. Re-tuning can restore the escalation rate $p$ and may fix most exit errors without any new calibration objective. It cannot move a zoom window that quantization has placed on the wrong region, however. That is why the diagnostic phase separates exit agreement from zoom-region overlap. It is also why re-tuning is built into the strongest dynamic baseline rather than treated as a straw man. If re-tuning alone closes the oracle-routing gap, the honest conclusion is that thresholds suffice and the top-K term is unnecessary. We would report that and narrow the contribution to the stage-precision split.

Why not just quantize a static ViT at native resolution? Uniform low-bit quantization lowers the cost of each operation but leaves the token count untouched, so attention cost still grows quadratically with patches. LF-ViT's 63% FLOP cut and its throughput doubling from 2601 to 5271 images per second on an A100 come entirely from spatial savings. Static quantization cannot reproduce those savings. The matched-latency comparison with Liu et al. and PMQ settles the question directly. If a static 4-bit model at equal latency matches our accuracy, the dynamic machinery is not worth its complexity.

Why not just use the existing all-pairs ranking-aware loss? This alternative could plausibly work as well. It already targets attention-order flips and may protect the top-K set as a side effect. Our bet is that spending calibration capacity on the full ordering is wasteful when only the set boundary matters. The swap ablation at matched calibration cost decides this. A tie favors the simpler existing loss, and we would say so.

Why not just keep the coarse stage in full precision and quantize only the fine stage? This removes misrouting by construction. Its cost is that the coarse stage runs on every image, so it dominates expected cost when $p$ is small, which is precisely LF-ViT's regime. The $(b_c, b_f)$ grid includes the 16/4 point. The cost equation predicts it loses on latency whenever $p$ is low, and the measurements will test that prediction.

Proposed Experiments

The main experiment is on ImageNet-1K. It uses LF-ViT and CF-ViT on DeiT-S and LV-ViT, with stage precisions drawn from the grid above. Every configuration reports top-1 accuracy, bit-operations, measured A100 throughput, and INT8/INT4 latency, alongside the three routing statistics.

The first decisive check is whether oracle routing closes a substantial share of the 4-bit accuracy gap. If it does not, misrouting is not the main problem and the method is moot. The second is whether routing-preserving calibration beats threshold re-tuning on zoom-region overlap and on final accuracy at matched $p$. The third is whether the best dynamic quantized configuration dominates static quantized DeiT-S and DeiT-B on the measured-latency frontier.

The secondary experiment repeats the comparison on CUB-200 and Stanford Cars at raised input resolution. There we expect misrouting to cost more and the method to help more. If the gain does not grow in this regime, that falsifies the story that the method helps by protecting fine-detail localization.