Certifying Inaudibility Frame by Frame in Streaming Speech Codecs

A speech codec for live calls has to decide, every 10 to 20 milliseconds, how many bits to spend on the frame it just captured. It cannot wait for future audio, and it cannot know in advance whether the network will deliver the packet. Psychoacoustics says the goal is not zero coding error. The goal is error that stays below the masking threshold, the level under which the ear cannot hear noise because louder nearby sound covers it. This proposal builds an online controller that adjusts a streaming neural codec's bitrate frame by frame. Over any single call, the fraction of frames whose coding noise rises above the masking threshold stays below a target, say 5%, with no assumption about the speaker, the noise, or the channel. The controller reuses online conformal prediction, a simple feedback rule that nudges a threshold up or down after each observed mistake. We apply it to a binary event that the ear actually cares about: "this frame's noise was audible in some frequency band."

This matters because current systems sit at two poorly matched extremes. Perceptual training losses make codecs and enhancers sound better on average, but they give no per-call promise and assume perfectly delivered, synchronized frames. Online conformal compression, by contrast, gives per-sequence guarantees for zero-delay links, but only for 0–1 losses on text tokens. Its authors name continuous distortion as the open problem. Our observation is that the masking threshold supplies the missing 0–1 loss. Once audibility is the event being controlled, the existing guarantee applies to speech essentially unchanged. The research question becomes whether the resulting bit savings and robustness are real on speech with distribution shift and bursty packet loss.

Core Claim

Define, for frame $t$, the noise-to-mask ratio $\mathrm{NMR}_{t,b}$ as the coding-noise energy in critical band $b$ divided by the masking threshold in that band. The threshold is computed causally from the current and past input only, so it includes simultaneous masking and post-masking but not pre-masking. A frame is audibly violated when

$$ v_t = \mathbb{1}\left[\max_b \mathrm{NMR}_{t,b} > 0\ \mathrm{dB} \ \text{ or packet } t \text{ was lost}\right]. $$

The encoder keeps a tolerance $\gamma_t$, measured in dB of allowed noise-to-mask ratio. At each frame it picks the fewest residual quantizer stages whose locally decoded output satisfies $\max_b \mathrm{NMR}_{t,b} \le \gamma_t$. This is the analysis-by-synthesis loop of classic perceptual coders, but with a moving target. After each frame it updates

$$ \gamma_{t+1} = \mathrm{clip}_{[\gamma_{\min},\gamma_{\max}]}\big(\gamma_t - \eta\,(v_t - \alpha)\big), $$

where $\alpha$ is the target audible-frame rate and $\eta$ a step size. A violation lowers the tolerance, which spends more bits; a run of clean frames raises it, which saves bits.

The standard online conformal argument then gives a per-sequence bound. Its form comes from telescoping the bounded updates. The bound is

$$ \frac{1}{T}\sum_{t=1}^{T} v_t \le \alpha + \frac{\gamma_{\max}-\gamma_{\min}+\eta}{\eta T}. $$

It holds for every input sequence, provided one condition is met: at $\gamma_{\min}$ the encoder has a fallback mode that is reliably violation-free when delivered.

We state two caveats up front. First, packet losses cannot be forced to zero by spending bits on refinement. The controller therefore also chooses how many coarse stages of frame $t$ to repeat redundantly in packet $t+1$, and violations caused by loss are known only after feedback. For delayed feedback we adopt the asynchronous, doubly-adaptive accounting of Prediction-Powered Communication with Distortion Guarantees. Second, the guarantee concerns a model-based audibility proxy. Whether that proxy tracks what listeners hear is an empirical question we must test, not assume.

Relation to Prior Work

The control rule is borrowed from Online Conformal Compression for Zero-Delay Communication with Distortion Guarantees. That paper wraps an autoregressive language-model predictor in online conformal prediction and reports 30% (synchronous) and 16% (asynchronous) bitrate reductions against a dropout-LLMZip baseline on text. It keeps outage below $\alpha + o(1)$ through a shift from Shakespeare to song lyrics. Their guarantee is limited to 0–1 losses on token prediction sets. We keep the guarantee mechanism but change the object being controlled. Instead of a prediction set over tokens, we control quantizer depth on audio, and instead of token outage, the event is a masking violation.

Three lines of work supply the perceptual side and motivate the gap. Development of a Psychoacoustic Loss Function for the DNN-Based Speech Coder trains a neural coder with a noise-to-mask loss weighted by perceptual entropy and beats AMR-WB at 13–24 kbps. It assumes synchronized, loss-free frames and offers no runtime control. A Perceptual Weighting Filter Loss for DNN Training in Speech Enhancement shows that a CELP-style weighting used only at training time gives +0.05 to +0.11 PESQ at zero inference cost. Its weighting is static, which suits our codec training but not our runtime controller. MUFFIN allocates codebooks by perceptual sensitivity in latent sub-bands. Its latent FFT needs multi-frame context, and it degrades on emotional speech in IEMOCAP. That degradation is precisely the kind of distribution shift a fixed allocation cannot react to and an online controller can.

Two papers constrain the design. Psychoacoustic Challenges of Speech Enhancement on VoIP Platforms shows that quality collapses when processing is cascaded with real networks: the PESQ effect moves from −1.872 to −2.777 with platform denoising, and to −3.382 for Google Meet over cloud with denoising. This is why channel loss must enter the controlled event rather than be ignored. On Real Time Coding with Limited Lookahead quantifies the cost of shrinking lookahead. That is why we drop pre-masking, which would need future samples, and accept a slightly conservative threshold.

Classical sequential rate-distortion theory takes a different route. Semidefinite Programming Approach to Gaussian Sequential Rate-Distortion Trade-Offs gives exact optimal real-time codes, but only for Gauss–Markov sources under squared error. We deliberately give up optimality for a distribution-free guarantee on a non-quadratic, perceptual criterion.

Research Program

We first build a causal, bitrate-scalable neural codec. It uses causal convolutions in the style of real-time Demucs (Défossez et al.), which runs on a 16 ms stride. Its residual vector quantizer is trained with random stage dropout, so that any prefix of stages decodes sensibly. The training loss combines the noise-to-mask objective of Byun et al. with multi-resolution spectral terms. We target a 20 ms frame with no lookahead beyond the current frame, giving under 25 ms algorithmic delay versus LPCNet's 65 ms. The top operating point adds a fine residual scalar layer as the fallback mode; we verify empirically that it is violation-free on clean delivery.

Second, we implement the causal masking model. It uses a Bark-band spreading function, absolute threshold of hearing, and post-masking from the previous two frames. We also implement a "wrong-metric" variant that replaces the noise-to-mask ratio with per-band SNR, used for ablation.

Third, we implement the controller with three parts: the refinement knob, the redundancy knob, and delayed-feedback accounting. We choose between refinement and redundancy with a simple rule. Redundancy is added only when the recent loss-attributed share of violations exceeds half of $\alpha$. We will report whether a learned allocation helps beyond this rule.

Fourth, we run the evaluation against the baselines below.

Baselines

The naive reference is the same codec at a fixed number of stages, sweeping bitrates. It defines the rate-versus-violation curve that any adaptive scheme must beat.

The strongest classical competitor is a per-frame greedy rate loop with a fixed tolerance $\tau$. For each frame it chooses the fewest stages with $\max_b \mathrm{NMR}_{t,b} \le \tau$. We tune $\tau$ offline on a calibration split so that the audible-frame rate on that split equals $\alpha$. This is exactly what an experienced audio-coding engineer would build. It shares our codec, masking model, latency, and compute, so it isolates the one thing we add: online adjustment of the tolerance. Beating it means two things. On in-distribution test data, it should match the realized audible-frame rate at equal or lower average bitrate. Under shift, its violation rate should exceed $\alpha$ while ours stays within the bound, at a bitrate penalty we report.

A second learned competitor is a constant-bitrate codec trained with the Byun et al. psychoacoustic loss at the matched average bitrate. No reported result for its exact PESQ or MUSHRA values was found on Lacuna; the paper reports only that it exceeds AMR-WB. As external reference points we report LPCNet at 1.6 kb/s with 65 ms delay (Valin and Skoglund). Its MUSHRA advantage over MELP is reported only qualitatively, so it serves as a latency and rate anchor, not as a quality target.

For channel robustness, the baseline is the greedy loop plus fixed-rate redundancy, tuned to the same average bitrate. All methods share training data, codec weights, and the same tuning budget on the calibration split.

The key ablation swaps the masking-based violation for an SNR-based violation at matched audible-frame rate. If listening quality does not differ, the psychoacoustic ingredient is not earning its place.

Why not just…?

Why not just use the per-frame greedy loop with a well-chosen tolerance? This is the real threat, and on stationary, in-domain speech it may tie with our method. Its weakness is that $\tau$ is calibrated on one distribution. When the input moves to emotional speech or noisy far-field speech, the frequency of hard frames changes, and nothing corrects the realized violation rate. This is the same failure mode online conformal compression fixed for text under the Shakespeare-to-lyrics shift. The shift experiments below settle the question. If the greedy loop's violation rate stays near $\alpha$ under shift, our contribution reduces to the loss-handling component, and we will say so.

Why not just train with a perceptual loss and send at a fixed rate? Training-time perceptual losses improve average quality: Zhao et al. report +0.05 to +0.11 PESQ, and Byun et al. report gains over AMR-WB. But neither offers runtime control or any per-call statement. They must either over-provision bits for the worst calls or accept unbounded violations on hard ones. Our method uses such a loss for the codec and adds the controller on top, so the comparison directly measures what the controller adds.

Why not just add forward error correction or use packet-loss concealment? Fixed redundancy is tuned for an average loss rate. Bursty Gilbert–Elliott losses either waste it in good periods or overwhelm it in bad ones. The VoIP measurements of Konan et al. show how badly fixed cascaded processing degrades over real networks. Our redundancy knob is driven by observed loss-attributed violations. The fixed-redundancy baseline at equal average bitrate tests whether this adaptivity matters.

Why not just use a dry/wet mix or a heuristic post-filter? Mixing in 5% of the input, as in Défossez et al., adds back noise uniformly across bands. It is an enhancement trick, not a rate controller, and is orthogonal to the bit-allocation question. We therefore exclude it as a baseline rather than argue against it.

Proposed Experiments

The codec is trained on LibriTTS. In-domain test sets are LibriTTS test-clean and TIMIT at 16 kHz. Shift test sets are IEMOCAP, where MUFFIN reports degradation, and noisy speech from DNS Challenge 2020 and Valentini. We also test concatenated "calls" that switch domains midway, which probes recovery time. Channels are i.i.d. erasure and Gilbert–Elliott bursty erasure at mean loss rates from 1% to 10%, with acknowledgement delays of 0, 2, and 5 frames.

Metrics are the realized audible-frame rate against $\alpha \in \{0.02, 0.05, 0.1\}$, average and 95th-percentile bitrate, and PESQ and STOI. A MUSHRA test compares our controller, the greedy loop, and the SNR-controlled ablation at matched bitrate.

The first decisive check is the guarantee itself. On every individual call, across all shifts and channels, the realized audible-frame rate must respect the bound above. Where it cannot, because loss exceeds what redundancy can repair within budget, the controller must report the infeasibility rather than fail silently.

The second check is efficiency. In-domain, our average bitrate should be no worse than the greedy loop's at equal violation rate. Under shift, the greedy loop should miss $\alpha$, while we pay a bitrate premium that we report honestly.

The third check is perceptual validity. MUSHRA and PESQ scores should rank systems in the same order as their audible-frame rates. In particular, the masking-controlled system should beat the SNR-controlled ablation at matched bitrate. If it does not, the masking proxy is not a valid target, which would be a useful negative result for any future perceptual certification work.

The fourth check is edge feasibility. We measure controller and masking-model overhead per frame and real-time factor on a laptop CPU, compared with the 0.6–0.8 real-time factor reported for causal Demucs. Unlike the language-model predictors in the conformal compression work, our controller is a scalar update and should add negligible compute.