When a neural audio codec (a network that compresses speech into a few discrete tokens per frame and regenerates a waveform from them) is pushed below a few kilobits per second, it fails in several different ways. It can muffle the voice, add hiss, drop syllables, change who seems to be speaking, or swap one sound for another. The usual evaluation reports one opinion score, and that score cannot tell these failures apart. The standard alternatives also break for this kind of codec. Sample-level metrics such as PESQ penalize a decoder that regenerates plausible audio that is not phase-aligned with the input. The existing multidimensional quality predictors were built for telephone impairments and never saw a generative codec.
This note proposes a reference-based diagnostic. Pass the original and the decoded speech through a frozen self-supervised speech model, a network pretrained on unlabeled audio whose layers are known to move from acoustics toward phonetics and meaning. Then measure how far apart the two signals are at every layer. The claim is that the shape of this "divergence-by-depth" profile separates the perceptual dimensions of codec damage. Shallow-layer divergence tracks coloration and noise. Lower-to-middle divergence tracks speaker and timbre loss. Middle-to-deep divergence tracks content errors. Because the profile is computed on learned representations rather than waveform samples, it should tolerate phase-free resynthesis. A small constrained map from the profile to human ratings on each dimension should transfer to codec families it was not fitted on.
Core Claim
The question is whether the layer hierarchy of a speech model, documented on clean speech, survives codec degradation well enough to serve as a decomposition of the damage. On clean speech the hierarchy is well documented. Speaker information concentrates in lower and middle layers of HuBERT, wav2vec 2.0 and UniSpeech-SAT, according to learnable layer weights in Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification. Early layers are sensitive to timbre (Baumann), phonetic information peaks mid-network (Glass et al.), and final layers over-specialize to the pretraining loss (Chi et al.). The research file lists the stability of this structure under low-bitrate compression as uncharacterized.
The idea is concrete. For a reference utterance $x$ and its decoded version $\hat{x}$, let $h_l(\cdot)$ be the frame sequence at layer $l$ of a frozen model with $L$ layers. We define
$$d_l(x,\hat{x}) = \frac{1}{|P|}\sum_{(t,s)\in P} \left(1-\cos\big(h_l(x)_t,\ h_l(\hat{x})_s\big)\right),$$
where $P$ is the dynamic-time-warping alignment path between the two sequences. The minimal-pair ABX work, Evaluating speech features with the minimal-pair ABX task, uses the same DTW-plus-cosine comparison as a model-free probe of representations, and we reuse its technique. We then normalize each $d_l$ by its value under perceptually transparent perturbations, such as re-encoding at a very high bitrate or sub-sample shifts. This puts the layers on a common scale. The resulting profile $D=(d_1,\dots,d_L)$ is mapped to each perceptual dimension $k$ by a nonnegative, sparsity-penalized linear read-out $\hat{y}_k = w_k^\top D + b_k$ with $w_k \ge 0$.
The nonnegativity and sparsity constraints are what make the method diagnostic. Each dimension must be explained by a small band of layers. The fitted bands can be checked against the clean-speech hierarchy and against controlled interventions.
Research Program
The work has three parts: a label set, the profile analysis, and controlled interventions.
The first part fills a gap no existing resource covers. There are no human ratings of neural codec output along separate perceptual dimensions. Speech quality evaluation of neural audio codecs ran a single degradation-category rating over Lyra V2, EnCodec, AudioCraft, AudioDec, DAC, Opus, EVS and LPCNet. It found distinct failure profiles that its scalar score could only describe in prose: EnCodec's bandwidth ceiling, AudioCraft's residual noise at 6 kbps, and gender-dependent behavior. We will run a crowdsourced listening test using the rating scales of the NISQA corpus (noisiness, coloration, discontinuity, loudness, and overall), following NISQA. We add two degradation-rated scales that generative codecs require: speaker/timbre fidelity to the reference and content fidelity. The file flags phonemic hallucination and timbre drift as dimensions missing from current frameworks. Codec-SUPERB shows that speaker verification degrades sharply under compression while recognition survives, which motivates keeping speaker as its own axis.
The rated material will be LibriSpeech and VoxCeleb1 utterances from the Codec-SUPERB speech pool. Each is passed through EnCodec, DAC, SNAC, SpeechTokenizer, AudioDec, Opus and EVS at bitrates spanning roughly 1 to 24 kbps. SNAC and SpeechTokenizer matter because their codebooks are structured by timescale and by semantics respectively. Truncating their codebooks therefore removes known kinds of information.
The second part computes profiles from WavLM, HuBERT and wav2vec 2.0 and fits the constrained read-out under leave-one-codec-family-out cross-validation. It then asks two questions. Do the selected layer bands differ across dimensions in the predicted order? Are they stable across the three backbones?
The third part uses interventions with known answers, requiring no listeners, to test whether the bands are causal markers or incidental correlates. Low-pass filtering and spectral tilt should move shallow layers. Additive noise at matched level should move shallow layers with a different band shape. Frame dropouts should register in the DTW path cost. Pitch and formant shifts should move lower-middle layers while leaving deep layers mostly unchanged. SpeechTokenizer decoded from its first codebook alone, which is distilled from HuBERT layer 9, should leave deep-layer divergence small while shallow and speaker-band divergence grows. If deep divergence grows instead, the "content" band is not what we think it is.
We also test the non-linear aggregation problem. Multidimensional evaluation and predicting overall speech quality showed that coinciding severe impairments break linear combination, and the file reports Möller and colleagues handling dominant single impairments by subtracting degradations from a maximum quality. We compare a linear combination of predicted dimensions against a dominant-impairment rule for predicting overall score on low-bitrate codecs, where compound damage is common.
Baselines
The naive reference point has two parts: overall-score-only prediction, and PESQ/STOI as the signal-level diagnostic codec papers already report. PESQ is expected to under-rate generative decoders. STOI is included because the Codec-SUPERB SLT benchmark (Codec-SUPERB @ SLT 2024) found it correlates above 0.8 in magnitude with recognition, verification and emotion outcomes. Both are computed on the same clips at no tuning cost. Beating them means higher per-dimension Pearson and Spearman correlation on held-out codec families.
The strongest existing multidimensional predictor is NISQA. On its own test sets it reports RMSE between 0.16 and 0.56 across the four dimensions, outperforming the double-ended DIAL model on most datasets (NISQA). We run it two ways: off the shelf, and retrained on our training codec families with the same data splits and a matched hyperparameter search budget. The file reports no NISQA result on neural codec output, and more generally no reported result found on Lacuna for per-dimension prediction on neural codec audio. Beating it therefore means lower RMSE and higher correlation on held-out codec families for coloration, noisiness and the two new scales.
The strongest representation-based competitor is a UTMOS-style model. UTMOS fine-tunes self-supervised models to predict MOS and reached utterance-level SRCC 0.897 on the VoiceMOS 2022 main track. We extend it with one learnable layer-weighting and one head per dimension, as in the speaker-verification layer-weighting setup, and train it non-intrusively on the same labels with the same backbone and tuning budget. This is the direct test of whether computing a reference divergence at each layer adds anything beyond letting a fine-tuned model choose its layers.
The Codec-SUPERB application vector serves as a diagnostic baseline. It consists of Whisper WER, ECAPA-TDNN speaker similarity, emotion2vec agreement, STOI and Mel distance, fed to the same constrained regression.
Two ablations isolate the key ingredient. The first uses one pooled layer instead of the full profile. The second uses an unconstrained multilayer perceptron on the profile, to measure what the interpretability constraint costs in accuracy.
Why not just…?
Why not just fine-tune an SSL MOS predictor with one head per dimension? This is the real threat. With enough labels, a UTMOS-style model with per-head layer weights may match our read-out on in-distribution codecs. The proposal's bet is transfer across codec families. A non-intrusive model must learn what "clean" sounds like from the training distribution. The reference profile only has to learn which layer bands correspond to which percept, which is closer to a property of the backbone than of the codec. The leave-one-family-out comparison settles this. If the fine-tuned model is within its confidence interval on held-out families, we will report that the layer profile adds interpretability but not accuracy, and the contribution narrows to the dataset and the intervention analysis.
Why not just use the Codec-SUPERB battery of downstream task scores as the diagnostic? It is already multi-part, and it degrades heterogeneously across codecs (Codec-SUPERB). However, its components are task accuracies, not percepts. Recognition WER is nearly unchanged at bitrates where listeners clearly hear coloration, and no component targets coloration or noisiness. The regression baseline on this vector measures the gap directly on the human-rated dimensions. The speaker-verification component may well match our speaker band, and we will report it if so.
Why not just ask an audio language model to describe the degradation? QualiSpeech fine-tuned SALMONN-7B to reason before scoring and reached PCC 0.721 for noise and 0.660 for overall quality. Those figures are below NISQA-class accuracy, and the paper notes that smaller open backbones lag closed models. It also treats the model as a black box and has no codec material. We include its model as a secondary baseline on our test set, but we do not expect it to be the binding competitor.
Proposed Experiments and Decisive Checks
Four checks decide the project.
The first is the intervention test. Each synthetic manipulation must move the layer band assigned to its dimension more than other bands, consistently across WavLM, HuBERT and wav2vec 2.0. If low-pass filtering and pitch shifting produce indistinguishable profiles, the decomposition fails and we stop.
The second is held-out-family accuracy on the new ratings. The profile read-out must beat retrained NISQA and match or beat the multi-head UTMOS-style model on coloration, noisiness, speaker fidelity and content fidelity, as measured by Spearman and Pearson correlation with bootstrap intervals over utterances and listeners.
The third is qualitative diagnosis of documented failures. The profile should attribute EnCodec's bandwidth ceiling to coloration and AudioCraft's 6 kbps residual noise to noisiness, matching the findings in Speech quality evaluation of neural audio codecs, and truncating SNAC's fine-scale codebooks should register as shallow damage.
The fourth is aggregation. On clips with two or more predicted severe dimensions, the dominant-impairment rule should predict overall ratings better than a linear sum, extending Berger et al.'s finding to neural codecs.
A side result, whatever the outcome, is the first characterization of how the self-supervised layer hierarchy behaves under low-bitrate coding, which the file identifies as an open problem.
References
- Large-Scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification
- Detecting Vocal Fatigue with Neural Embeddings
- Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems
- Audio Albert: A Lite Bert for Self-Supervised Learning of Audio Representation
- Evaluating speech features with the minimal-pair ABX task: analysis of the classical MFC/PLP pipeline
- Speech quality evaluation of neural audio codecs
- NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets
- Codec-SUPERB: An In-Depth Analysis of Sound Codec Models
- SNAC: Multi-Scale Neural Audio Codec
- SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models
- Multidimensional evaluation and predicting overall speech quality
- Codec-Superb @ SLT 2024: A Lightweight Benchmark For Neural Audio Codec Models
- UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022
- QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions