Codebook Ladders: Free Supervision for Perceptual Metrics of Neural Audio

Neural audio codecs compress sound into several stacked codebooks. A codebook is a learned table of vectors, and each audio frame is stored as a short list of table indices. In the common "residual" design, each codebook encodes what the previous ones missed. Decoding with only the first $k$ codebooks therefore gives a ladder of reconstructions, from coarse to complete.

One recent study found that this ladder fools standard metrics. Log-spectral distance stops improving after about 3 of 9 codebooks, yet blind listeners strongly prefer all 9, because the upper codebooks remove a harshness that spectral distances cannot see (Geometric Iterative Retrieval for Neural Audio Codec Resynthesis). We propose to turn this failure into a resource. Every clip run through any residual codec yields many "rung $j$ sounds better than rung $i$" pairs at no annotation cost. These pairs isolate exactly the kind of quality difference that objective metrics miss.

We will train a quality metric on these ladder pairs across codecs and across speech and music. We then ask whether it predicts human ratings on codecs and generated audio it never saw, including codecs with parallel subspace codebooks, which have no ladder at all. In two sentences: codebook truncation is a cheap, ordered source of codec-artifact severity. A metric trained to respect that order may track human judgment of neural audio better than metrics trained on hand-designed distortions or on a few hundred human ratings, especially for music, where no validated metric exists.

Core Claim

Our claim has two parts, and both can fail. First, the ladder order (more residual codebooks sounds better) holds across codecs and domains for rung gaps above some size. Second, a metric trained with a ranking objective on these orders transfers beyond the ladders. It should correlate with listener ratings when comparing different codecs at matched bitrate, codecs with parallel subspace codebooks, and outputs of token-based generators.

The research file shows that perceptual metrics for neural codecs have been evaluated, and that the residual-layer divergence has been observed. It does not show anyone using the codec's own codebook hierarchy as supervision for a metric. That use is the new part.

Prior Work and Positioning

Three papers motivate the gap.

Two papers supply method.

Two papers constrain the design.

  • The music evaluation literature shows that distribution-level scores such as Fréchet Audio Distance reward spectral texture over musical coherence (FreeSound Loop benchmark). So we will not use them as per-sample references.
  • SongEval and CMI-RewardBench show that musical aesthetics vary independently of signal fidelity. We therefore scope our metric to codec and synthesis artifact quality, not musicality, and we treat aesthetic predictors as a separate axis rather than a competitor.

One more finding cuts against relying on residual hierarchies for analysis. HiFi-Codec and the retrieval paper report that residual codecs pack most structure into the first codebook. This is a reason to test whether ladders from different codec families, such as grouped, band-split and plain residual, teach the same notion of quality.

Research Program

Building the ladders. We tokenize speech (LibriTTS, Common Voice, WenetSpeech) and music (MTG-Jamendo, FMA) with several residual codecs: DAC, EnCodec, HiFi-Codec and a MUFFIN-style band-split codec. We decode every prefix of codebooks. Where public checkpoints are unavailable, we retrain at published configurations.

Validating the order. Before any training, we check the assumption the method rests on. Using a MUSHRA protocol (listeners rate several versions of a clip on a 0–100 scale against a hidden reference), we collect ratings on a stratified sample of rungs, codecs and domains. This tells us for which rung gaps the order is reliable. Only gaps that pass are used for training.

Training the metric. The metric $f$ maps a clip, optionally with its reference, to a scalar. We train it with a margin ranking loss:

$$ \mathcal{L} = \sum_{(x_i, x_j)} \max\!\big(0,\ m(j-i) - f(x_j) + f(x_i)\big) $$

Here $x_i$ and $x_j$ are rungs $i < j$ of the same clip from the same codec, and $m(j-i)$ is a margin that grows with the gap between rungs. We fit $m$ to the human-validated gap curve, so that larger perceived gaps demand larger score differences. We train two variants, one with access to the reference and one without, so we can compare against both intrusive and non-intrusive baselines.

Blocking the shortcut. The obvious risk is that $f$ learns "count the codebooks" by detecting high-frequency energy or bitrate signatures, not quality. We take three countermeasures:

  1. Hold out whole codec families at test time.
  2. Add cross-codec pairs at matched rung. These are ordered only by the small human-labeled subset; they are never assumed.
  3. Run an adversarial probe where broadband noise or hiss is added to low rungs, raising high-frequency energy while making the audio worse.

Diagnostic use. Once validated, the metric can attribute quality change to each additional codebook in residual and grouped codecs. For parallel product-quantized codecs, we replace one sub-codebook at a time with its nearest-neighbor perturbation. This answers the per-codebook questions left open by the black-box codec benchmark. We regard this as secondary: it is only worth reporting if the metric passes the transfer tests.

Baselines

Naive reference. The do-nothing reference is the set of spectral distances already used to judge ladders: log-spectral distance and multi-resolution mel distance. Per Geometric Iterative Retrieval, these saturate at 3 codebooks. We compute them on identical clips and report rank correlation with every human test set. Any learned metric that fails to beat them on held-out codecs has learned nothing useful.

Strongest existing speech metrics. These are SCOREQ and PESQ. On ODAQ, Evaluating Objective Speech Quality Metrics for Neural Audio Codecs reports Pearson correlation with MUSHRA of 0.937 for SCOREQ on clean speech and 0.903 for PESQ on noisy speech. We run both unchanged on all our speech and music test sets. On ODAQ we use the paper's exact protocol, so the comparison sits on its reported numbers.

Beating them has two parts. On speech, our reference-based variant should at least match 0.937 on ODAQ clean speech within the confidence interval; ODAQ has only 11 clips per condition, so intervals will be wide. On our larger new speech MUSHRA set, it should exceed SCOREQ significantly. On music, it should exceed both by a clear margin on held-out codecs. We found no reported result on Lacuna for any objective metric's correlation with human ratings of music codecs, so our music test set establishes the first such number.

General-audio and aesthetic metrics. ViSQOL v3 in audio mode is included as the established general-audio intrusive metric. Audiobox-Aesthetic and a SongEval-trained predictor are included only to show that aesthetic scores measure a different axis. SongEval reports Pearson 0.912 against its own annotations, but those annotations target song aesthetics, not codec artifacts.

Ablations isolating the key ingredient. The ingredient is the source of the ordered pairs, so the ablations hold the network, data volume and compute fixed and change only that source:

  • pairs from standard synthetic distortions (noise, filtering, clipping, compression) at matched severity levels;
  • ladder pairs with an L2 regression to rung index instead of ranking, which tests SCOREQ's argument within our setting;
  • ladder pairs from a single codec family only, which measures how much codec diversity contributes.

All variants receive the same tuning budget and model-selection set. Model selection uses human ratings on a validation split disjoint from all test clips.

Why not just…?

Why not just use SCOREQ or PESQ, which already correlate above 0.9 with listeners? This is the most serious threat, and the first experiment addresses it. If SCOREQ already orders ladder rungs the way listeners do on music and on held-out codecs, the training step adds nothing. In that case the project reduces to the diagnostic study, and we will say so. There are reasons to doubt it. Both numbers come from speech on 11-clip ODAQ subsets. SCOREQ was trained on speech degradations. Nothing in the file shows either metric tracking the harshness-versus-smoothness difference on music that the retrieval paper's listeners heard.

Why not just collect more human ratings and fine-tune a predictor on them? Human ratings are the gold standard, and we do collect them for validation and test. But they cost orders of magnitude more per pair than ladders. The SCOREQ analysis also shows that L2-fit predictors on modest labeled sets generalize poorly to new neural artifacts. We include a "same budget spent on human labels" comparison by fine-tuning a pretrained audio encoder on our human training split alone. If it matches the ladder-trained metric on held-out codecs, ladders are not worth their complexity.

Why not just train on synthetic distortions, as existing learned metrics do? Synthetic distortions do not produce what neural decoders produce. Decoders invent plausible high-frequency texture, and the codec metric benchmark shows that metrics built on classical degradations misjudge these artifacts. The matched synthetic-pair ablation in the Baselines section tests this directly. If it performs as well as ladder pairs, our central claim is false.

Why not just learn a bitrate detector, since that is all a ladder teaches? This is the failure mode we most expect. The adversarial hiss probe and the matched-rung cross-codec pairs are designed to catch it. A detector of high-frequency energy will rate hiss-boosted low rungs above clean high rungs, while listeners will not. If our metric fails these probes, we will report it as a shortcut learner regardless of its correlation numbers.

Proposed Experiments and Decisive Checks

New listening tests. The program depends on a new MUSHRA study with trained listeners, in three parts:

  • Ladder validation over DAC, EnCodec, HiFi-Codec and the band-split codec, on speech and music clips drawn from LibriTTS, Common Voice, MTG-Jamendo and FMA.
  • A held-out transfer set: PQ-VAE reconstructions, cross-codec comparisons at matched bitrate, and hiss-perturbed low rungs.
  • A generative set, where coarse tokens are sampled from a token language model trained on DAC tokens, plus VALL-E-style speech from MUFFIN tokens, and then decoded. This is the generative regime that the retrieval paper left unexamined.

ODAQ serves as an external check on speech.

Primary metric. We report Spearman and Pearson correlation between each metric and mean listener scores, per test set, with bootstrap confidence intervals.

What would establish or refute the claim. Three results together establish it:

  1. The ladder-trained metric beats SCOREQ, PESQ, ViSQOL and spectral distances on music and on held-out codec families.
  2. It matches SCOREQ on ODAQ clean speech.
  3. It ranks hiss-perturbed rungs below clean higher rungs, in agreement with listeners.

Two results refute it. If the synthetic-distortion ablation or the human-label-only model matches it on held-out codecs, the ladder signal is not doing the work. If its correlation on generated audio collapses to the level of spectral distance, ladders teach codec-specific cues that do not transfer.

Secondary analysis. Only if the primary checks pass, we use the metric and listeners together to locate which codebooks or subspace codebooks introduce harshness in residual versus parallel designs. That directly addresses the open comparison between hierarchical and parallel factorizations.