Large speech models can be made cheaper by deleting audio frames that carry little new information, such as silence and the steady middle of a vowel. The best current method decides which frames to delete using attention scores from the first two or three layers. It can drop roughly half the frames with almost no loss in transcription accuracy. That evidence comes from clean English audio. Speech heard through a cochlear implant (CI), a device that replaces the inner ear with 16 to 22 electrodes, is very different. Neighbouring frequencies blur together, pitch is mostly gone, and the brief consonant cues that separate /b/ from /d/ become faint smears.
This note proposes that what counts as a "redundant" frame depends on what the listener can resolve. When input is degraded in this way, the layer at which a model can tell important frames from unimportant ones moves deeper. A pruner that works for clean speech will therefore throw away the weak consonant transitions and, in Mandarin, the tone-bearing vowel frames that CI users most need. The fix we propose is to prune in two stages: delete silence early, and delay all within-speech pruning until the layer where consonant (and tone) information has formed on degraded input.
The question matters practically. Efforts to put foundation-model processing in the CI signal path, such as AV-HuBERT-based enhancement before a CI vocoder and neural replacements for the clinical coding strategy, will only run on wearable hardware if their compute shrinks. Frame pruning is the main tool for that, and nobody has checked whether it is safe on the kind of signal these systems handle.
Core Claim
The claim has two parts. The first is a measurable regularity. Under CI-like degradation (a 16-channel vocoder with simulated current spread), the layer at which linear probes recover consonant place of articulation and lexical tone moves deeper than it is for clean speech. Attention-based importance scores taken from early layers then systematically underrate frames containing smeared transitions.
The second part is a method that follows from the regularity. The pruner removes non-speech frames at an early layer, where signal-envelope energy survives degradation and silence detection is still reliable. It then removes within-speech frames only at a depth $L^*$. $L^*$ is calibrated on degraded data as the shallowest layer where the consonant-place probe (plus a tone probe, for tonal languages) reaches a fixed fraction of its peak accuracy.
The method differs from existing work in one specific way: pruning depth becomes a function of input degradation and language, rather than a constant tuned on clean English.
Grounding in Prior Work
Two results supply the method. Early Attentive Sparsification scores tokens by cross-frame attention within the first three Whisper layers and drops 40–60% of them without retraining. We reuse its scoring, but question its fixed early depth, and it was never tested on degraded or tonal input. The phonological-arithmetic study [b]=[d]-[t]+[p] ([b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic) shows that self-supervised models encode phonological features as linear directions in their internal representations, and that this structure appears in middle and late layers. Vowels resolve earlier than consonants because consonants need wider temporal context. We use its probing approach to locate $L^*$, and its vowel/consonant ordering is the mechanism behind our prediction.
Three lines of CI work motivate the regime. Biological and electrical spectral smearing blurs adjacent frequencies (Jinbo et al.; MacIntyre et al.). Consonant place is the first contrast to fail for CI users (Medina et al.). Hines et al. found that clean timing cues do not help once spectral resolution is lost. Taken together, these suggest that under degradation a model must integrate over more time and more depth before a consonant becomes identifiable, which is the condition under which early pruning should fail.
Two CI systems motivate the application. ElectrodeNet replaces the clinical Advanced Combination Encoder with small networks and transfers from Mandarin to English. Lai et al. place an English-pretrained AV-HuBERT before a 16-channel vocoder and transfer to Mandarin. Both process every frame, and neither asks which frames matter.
Cross-lingual work constrains our predictions. Millet and Dunbar show that self-supervised models act as "universal listeners" without native-language narrowing, so we expect the depth shift for consonants to hold across languages. Tone is the exception. CI simulations show that Mandarin tone recognition depends on temporal fine structure that vocoders remove (Contributions of temporal fine structure cues to Chinese speech recognition in CI simulation). The residual tone cues sit in envelope shape and duration across vowel frames, exactly the frames an English-calibrated pruner treats as steady and disposable.
One honest limitation applies throughout. Vocoders are not electrical hearing: Kong et al. found that CI users readily tell vocoded speech from natural speech. Our claims concern models processing CI-like signals, which is the setting where these on-device systems would run. They are not claims about patient perception.
Research Program
We first build a degradation ladder. It runs from clean speech, through a 22-channel noise vocoder, to a 16-channel noise vocoder, and finally to a 16-channel vocoder with current spread simulated by broadened synthesis filters. We apply it to TIMIT (English, phone-aligned), LibriSpeech, TMHINT (Mandarin), and the German portion of Multilingual LibriSpeech as a non-tonal second language.
On each condition we run per-layer linear probes on Whisper-large-v3 and distil-whisper (the models EAS reports on), and on HuBERT for comparison with the phonological-arithmetic results. The English probes predict consonant place, manner, and voicing from TIMIT alignments. The Mandarin probe predicts lexical tone, using TMHINT labels derived from transcripts through a pinyin lexicon and forced alignment. The output is a curve of probe accuracy against layer for each degradation level. From each curve we read off $L^*$, the shallowest layer at which accuracy reaches 90% of that curve's peak.
In the same runs we record which frames EAS drops. We then measure the fraction of phone-aligned consonant-transition frames and tone-bearing vowel frames that are removed, relative to their share of all frames.
Next we implement the two-stage pruner as an inference-time modification with no retraining, matching EAS's setting. Stage one runs at layer 2 and only removes frames whose attention importance falls below a threshold calibrated to the non-speech regions of the calibration set. Stage two runs at $L^*$ and applies EAS-style top-K selection to the remaining frames. The total keep fraction is swept to trace accuracy against real-time factor, which is wall-clock processing time divided by audio duration. $L^*$ is calibrated once per degradation level and language on a held-out calibration split, never on test data.
Baselines
The naive reference point is the unpruned model on each degraded condition. It sets the accuracy ceiling and the real-time factor of 1.0× speedup that every method must improve on.
The second reference is fixed-rate decimation, keeping every second or fourth encoder frame. It uses the same models and data with no tuning, and it shows how much content-aware selection adds when consonant cues are already weak.
The strongest competitor is EAS itself, Early Attentive Sparsification. It is the only method in the file that prunes encoder tokens of a pretrained speech model without retraining, and on clean LibriSpeech it reports keeping at least 99% of baseline accuracy while dropping 40–60% of tokens. It reaches 1.24× speedup on whisper-large-v3 and 1.60× on distil-whisper, and it becomes unstable above 90% sparsity. No reported result was found on Lacuna for EAS on vocoded speech, TMHINT, or any tonal language.
We run EAS at its published early-layer setting with the same models, the same keep-fraction sweep, and the same calibration split for its threshold. Beating it means matching its accuracy retention on clean speech (within 1% relative). On the 16-channel-with-spread condition, it means either holding at least 99% of unpruned accuracy at a keep fraction where EAS falls below that, or achieving equal accuracy at a lower real-time factor.
Three ablations isolate the key ingredient. The first applies single-stage EAS at every layer from 1 to the final layer, with the chosen layer tuned per condition by grid search on the calibration split. This is the strongest possible "just move it deeper" competitor. The second is the two-stage pruner with $L^*$ calibrated on clean rather than degraded data. It isolates the claim that the depth must follow the degradation. The third is the two-stage pruner with random within-speech selection at $L^*$. It checks that the attention scores at $L^*$, and not merely the delay, carry the benefit.
Why not just…?
Why not just run EAS at a deeper layer? This is the most serious threat. A single prune at a well-chosen middle layer may keep accuracy on degraded speech as well as our method does. Its weakness is cost: every early layer still processes all frames, including silence, which is often a large share of an utterance. The per-condition layer sweep in the ablations settles the question directly. If single-stage pruning at the best layer matches the two-stage pruner on both accuracy and real-time factor, the method adds nothing beyond a tuning rule. The contribution then reduces to the measured depth shift and a recipe for predicting the right layer from probes without a test-time sweep. We would report it that way.
Why not just fine-tune the model on vocoded speech so early pruning works again? It might. Training could teach early layers to sharpen smeared transitions, and that would be an informative result in itself. We test it by fine-tuning Whisper on vocoded LibriSpeech and AISHELL-free Mandarin data drawn from the TMHINT training split, then re-measuring both the probe curves and EAS loss. If $L^*$ moves back to early layers after fine-tuning, the depth shift is a property of models never exposed to degradation rather than a lasting constraint. That limits the claim to frozen foundation models, which is still the case for both CI systems cited above. The fine-tuning route also costs a separate model per degradation profile, which matters given the large anatomical differences between CI users (Vandersteen et al.).
Why not just use a duration-predicting decoder? Token-and-Duration Transducers skip frames during decoding and report 2.19× speedup on LibriSpeech and 2.82× on German. However, the encoder still processes every frame, and the encoder dominates cost in Whisper-scale models. The two approaches are complementary rather than competing. We report the duration-predicting decoder only as a reference for how much decoder-side skipping alone gives on degraded input, without claiming to beat it.
Proposed Experiments and Decisive Checks
The first decisive check is the probe curve. On TIMIT, the consonant-place $L^*$ for the 16-channel-with-spread condition must sit at least two layers deeper than for clean speech, while the vowel $L^*$ moves little. If the curves do not shift, the central hypothesis is false and the method has no motivation.
The second check confirms the failure mode. At matched keep fractions of 40–60%, EAS on degraded LibriSpeech and TIMIT should lose more than its clean-speech 1%. The extra errors should concentrate on consonant-place confusions, measured by phone recognition on TIMIT, and consonant-transition frames should be dropped more often than their share of all frames.
The third check is cross-lingual. On TMHINT we compare character error rate and tone error rate in three settings: EAS, the two-stage pruner with $L^*$ calibrated on English, and the two-stage pruner with $L^*$ calibrated on Mandarin. We predict that English calibration protects consonants but still drops tone-bearing vowel frames, and that only the tone-aware $L^*$ restores tone accuracy. German serves as a non-tonal control, where English calibration should transfer.
The final check is the accuracy against real-time-factor frontier on all four degradation levels. The two-stage pruner should lie at or above the best single-stage layer from the sweep. It should also lie strictly above the EAS setting on degraded input, while staying within 1% relative of EAS on clean input.
All results are reported per degradation level, so a reader can see where along the ladder from clean to CI-like input the safe pruning depth begins to move.
References
- Early Attentive Sparsification Accelerates Neural Speech Transcription
- [b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmetic
- A hearing impairment simulation method using audiogram-based approximation of auditory charatecteristics
- Effects of spectral degradation on the cortical tracking of the speech envelope
- Development of voicing categorization in deaf children with cochlear implant
- Improved Speech Intelligibility with a Chimaera Hearing Aid Algorithm
- ElectrodeNet—A Deep-Learning-Based Sound Coding Strategy for Cochlear Implants
- Leveraging Self-Supervised Audio-Visual Pretrained Models to Improve Vocoded Speech Intelligibility in Cochlear Implant Simulation
- Do self-supervised speech models develop human-like perception biases?
- Contributions of temporal fine structure cues to Chinese speech recognition in cochlear implant simulation
- Cochlear-implant Listeners Listening to Cochlear-implant Simulated Speech
- In situ validation of a parametric model of electrical field distribution in an implanted cochlea
- Efficient Sequence Transduction by Jointly Predicting Tokens and Durations