Many on-device speech recognizers produce text twice. A fast encoder listens with no look-ahead and emits words immediately. A slow encoder then sits on top of the fast encoder's internal features, looks a few seconds into the future, and revises the transcript. This design is called a cascaded encoder. To fit these systems on phones, engineers shrink their weights from 32-bit floats to 8-, 4- or 2-bit integers. The standard way to decide which layers get which bit-width is to measure how much each layer hurts the loss when rounded. That method assumes one network with one output. A cascade has two outputs, and they value the same layer differently. The fast encoder's rounding noise reaches the streaming transcript directly. By the time it reaches the final transcript, the slow encoder has had seconds of future audio with which it may smooth that noise out.
We propose to treat quantization sensitivity in a cascade as a pair of numbers per layer, one for the streaming pass and one for the final pass. We then allocate bits with a budget that protects streaming accuracy explicitly, instead of folding both passes into one score. We also propose quantizing the one tensor unique to cascades: the buffer of fast-encoder features held in memory for the slow encoder's look-ahead. It is stored at very low precision, and the slow encoder is trained to read it that way. The central empirical bet is that the two sensitivity rankings diverge enough to matter. If they do not, the idea collapses into existing single-network methods, and we say in advance how we would detect that.
Core Claim
Given the same average bit budget, a pass-aware allocation gives a better trade-off between model size, first-pass word error rate (WER) and final WER than single-network sensitivity methods applied to the whole cascade. The allocation keeps a separate first-pass constraint. It is paired with a low-bit look-ahead buffer that the slow encoder learns to read. We expect the gain to appear exactly where uniform 4-bit stops being free, which means average budgets below 4 bits and parameter-dense models.
Where This Sits
The work closest to ours is 4-bit Conformer with Native Quantization Aware Training for Speech Recognition. It motivates our design and supplies our training machinery.
- What it did. It quantized a production causal plus look-ahead dual-encoder model using integer arithmetic during training ("native QAT", quantization-aware training). It found attention and boundary layers to be sensitive. It then used a hand-picked rule: early layers at 4-bit, later layers at 8-bit.
- What it did not do. It never measured how sensitivity differs between the causal and non-causal stages. It also reported that 4-bit activations degraded badly.
Lossless 4-bit Quantization of Architecture Compressed Conformer ASR Systems on the 300-hr Switchboard Corpus supplies a measurement technique we reuse. It scores each layer by the KL divergence of the output distribution when that layer alone is quantized, then assigns 2 to 8 bits. It did this only on a single offline encoder, with one output to protect.
Three further papers supply the allocation machinery we generalize from one loss to two:
- HAWQ ranks layers by the top Hessian eigenvalue normalized by parameter count.
- APTQ uses Hessian traces over attention projections and reports beating heuristic block schemes.
- GACT shows that quantization variance adds across tensors and solves bit assignment as an integer linear program.
On the architecture side, Streaming parallel transducer beam search with fast slow cascaded encoders supplies the model we compress. It uses Emformer layers with a 160 ms fast segment, a slow encoder with up to 6.4 s of look-ahead, and a 13+7 layer split. That look-ahead is what makes the buffer worth compressing.
A Unified Cascaded Encoder ASR Model for Dynamic Model Sizes constrains the framing. It already gets footprint and power savings in cascades through layer selection at floating point, so our gains must be orthogonal to depth reduction rather than a substitute for it.
Research Program
Step 1: measure a two-pass sensitivity profile. For layer $\ell$ quantized to $b$ bits, let $\Omega_{\ell,1}(b)$ be the increase in first-pass loss and $\Omega_{\ell,2}(b)$ the increase in final-pass loss. We estimate each two ways:
- directly, by quantizing that layer alone and measuring the per-pass KL shift on held-out LibriSpeech, following Li et al.;
- through a Hessian-trace approximation computed separately against each pass's loss, following HAWQ and APTQ.
Layers in the fast encoder receive both numbers. Slow-encoder layers affect only the final pass, so their $\Omega_{\ell,1}$ is zero by construction.
Step 2: a hypothesis about the fast encoder. We expect fast-encoder layers to show $\Omega_{\ell,2} < \Omega_{\ell,1}$. The reason is that rounding noise in causal features is largely uncorrelated across frames, and bidirectional attention over a long window averages it down. The ratio should grow with look-ahead length, which we will vary directly.
Step 3: allocate bits under a first-pass constraint. With $b_\ell$ the bit-width of layer $\ell$, $n_\ell$ its parameter count, $B$ the total bit budget and $\epsilon$ a tolerated first-pass loss increase, we solve
$$ \min_{\{b_\ell\}} \sum_\ell \Omega_{\ell,2}(b_\ell) \quad \text{s.t.} \quad \sum_\ell \Omega_{\ell,1}(b_\ell) \le \epsilon, \quad \sum_\ell n_\ell b_\ell \le B, $$
as an integer linear program over $b_\ell \in \{2,4,8\}$. The additive form rests on the same independence argument GACT proves for quantization variance. We will check it empirically by comparing predicted and measured loss for sampled joint configurations. The constraint form matters because it lets a product team fix streaming quality and spend every remaining bit on final accuracy.
Step 4: compress the look-ahead buffer. The fast-encoder output is stored for seconds at a time so the slow encoder can look ahead. The streaming decoder reads it at 8-bit. The slow encoder reads a 4- or 2-bit copy and is fine-tuned with native QAT on that copy. Ding et al.'s activation failure concerned activations throughout the network. Here only one tensor is quantized, and its reader has future context, so we treat the failure as an open question rather than a ruling.
Step 5: fine-tune the whole configuration. Every allocated model is fine-tuned with native QAT under the joint two-pass loss.
Baselines
Naive reference. The unquantized float cascade gives the accuracy ceiling. Uniform 8-bit post-training quantization gives the cheapest deployable point. Both are run on the identical trained checkpoint.
Uniform 4-bit with native QAT. This is the strongest simple competitor, and on LibriSpeech it is formidable: Ding et al. report a 7.7x size reduction (472 MB to 61 MB) with zero WER regression for a large Conformer. That result comes from a single large model, not a cascade. For a quantized cascaded encoder on LibriSpeech, no reported result was found on Lacuna. We will run uniform 4-bit and uniform 3-bit-average schemes with the same QAT epochs as our method.
Heuristic mixed precision. We run Ding et al.'s rule of 4-bit for early layers and 8-bit for later ones at matched average bits. As a second variant, we pin the first, last and attention layers to 8-bit, reflecting their sensitivity findings.
Strongest existing allocator. This is the Li et al. KL allocation, and a HAWQ-style Hessian allocation, applied to the cascade as one network with the summed two-pass loss. The reported point for this line of work is on Switchboard. Li et al. report a Hub5'00 average WER of 11.3% against an 11.1% float baseline at 16.2x total compression. That figure includes architecture search, so we compare only the quantization stage.
Ablations isolating the key ingredient:
- allocation by $\Omega_{\ell,2}$ alone;
- allocation by $\Omega_{\ell,1}$ alone;
- the summed-loss allocation with its mixing weight tuned on dev data, using the same number of dev evaluations our $\epsilon$ sweep uses;
- the buffer quantized without slow-encoder retraining;
- random allocation at matched budget.
All methods share the same checkpoint, calibration data, QAT schedule and model-selection budget.
What counts as beating a baseline. We beat a baseline if, at equal model size, we lower final WER with first-pass WER no worse. Either direction must hold on both test-clean and test-other, across three seeds.
Why not just…?
Why not just use uniform 8-bit post-training quantization? It is lossless and universally supported, but it caps compression at 4x relative to 32-bit floats. Ding et al. needed sub-8-bit to reach 5x to 7.7x for on-device dual encoders. Our uniform 8-bit run will locate this point on the size curve. If a product needs only 4x, this proposal is unnecessary, and we will say so.
Why not just quantize everything to 4-bit with native QAT? On large LibriSpeech models this appears lossless, which is a real threat. Our method helps only where uniform 4-bit breaks. Ding et al. saw that breakage at production scale, with 4-bit post-training quantization exceeding 100% WER on the 400,000-hour model, but we cannot access that data. We therefore test in the regime reachable on LibriSpeech: smaller, parameter-dense cascades and average budgets of 3 bits and below. If uniform 4-bit QAT matches us there too, our claim fails for academic-scale deployment.
Why not just apply HAWQ or KL allocation to the summed loss? This is the most serious alternative, because a tuned mixing weight might approximate the constraint. We settle it in two ways. First, before any allocation, we compute the rank correlation between the first-pass and final-pass sensitivity orderings of fast-encoder layers. If that correlation exceeds about 0.9, the two passes agree, and single-score allocation is sufficient by construction. Second, the tuned-weight ablation above gets an equal dev-evaluation budget.
Why not just distill a smaller cascade instead? Multi-stage Progressive Compression of Conformer Transducer reports 10% and 7.5% relative WER degradation at 64% parameter reduction, and 24% at 81%. It also leaves arithmetic in floating point. We treat distillation as complementary: we will quantize a distilled cascade too, as Li et al. did with factorized models.
Proposed Experiments
Model and data. We train an Emformer fast/slow cascade following Mahadeokar et al. at two sizes, on LibriSpeech 960 h. As a second domain we use Switchboard 300 h, evaluated on Hub5'00, which also allows comparison with Li et al.
Metrics:
- first-pass WER and final WER;
- average emission delay (Mahadeokar et al. report it nearly unchanged for the float cascade, so it is a regression check);
- model bits;
- peak memory of the look-ahead buffer;
- real-time factor on a mobile-class CPU.
For real-time factor, sub-byte values are stored in 8-bit containers. Both Li et al. and APTQ note this limitation, so we report memory traffic separately from compute time.
Decisive checks. Each is stated with the outcome that would falsify our claim.
- Divergence test. The profile study must show fast-encoder final-pass sensitivity falling below first-pass sensitivity, with the gap widening as look-ahead grows from 0.9 s to 6.4 s. A flat ratio would falsify the error-smoothing hypothesis.
- Additivity test. Predicted and measured loss for sampled joint configurations must correlate strongly enough for the integer program to be trusted.
- Pareto test. At 3-bit average, the pass-aware allocation must beat the tuned summed-loss allocation on final WER at matched first-pass WER.
- Buffer test. A 2- or 4-bit look-ahead buffer with slow-encoder QAT must cost well under the deliberation gains the cascade exists to deliver. Li et al. report those as 5% and 3% relative WER reduction on test-clean and test-other. If buffer quantization costs a comparable amount, it erases the reason to run a second pass, and we will report that as a negative result.
References
- 4-bit Conformer with Native Quantization Aware Training for Speech Recognition
- Lossless 4-bit Quantization of Architecture Compressed Conformer ASR Systems on the 300-hr Switchboard Corpus
- HAWQ: Hessian AWare Quantization of Neural Networks With Mixed-Precision
- APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models
- GACT: Activation Compressed Training for Generic Network Architectures
- Streaming parallel transducer beam search with fast slow cascaded encoders
- A Unified Cascaded Encoder ASR Model for Dynamic Model Sizes
- Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition