Neural audio codecs turn sound into a short stack of discrete codes per frame. They use residual vector quantization (RVQ): the first code captures a coarse approximation, and each later code encodes what the previous ones missed. Read top to bottom, a frame's stack of codes is a path in a tree: a root, one branch per first-layer code, one sub-branch per second-layer code, and so on down nine levels in the Descript Audio Codec (DAC). Count how many distinct branches a corpus of frames occupies at each depth and you get the tree's profile. Analytic combinatorics has studied that quantity in detail for digital search trees and tries. The central result is that the profile does not grow smoothly. It passes through sharply different regimes, near-complete, transitional, and nearly unique, and the switches happen at depths set by the ratio of depth to $\log n$ and by the source entropy, where $n$ is the number of stored strings.
This note proposes to use that theory to explain, and then measure, a gap that standard audio metrics cannot see. Geometric Iterative Retrieval for Neural Audio Codec Resynthesis found that log-spectral distance (LSD, a frame-by-frame comparison of spectra against a reference recording) stops improving after three RVQ layers. Listeners, by contrast, keep preferring outputs refined through all nine. The hypothesis is that this plateau sits where the codec tree leaves its transitional regime. Past that depth, codes are nearly unique per frame and cannot be predicted from coarse context, so matching the reference stops being the right target. What matters there is whether the distribution of deep codes is right. A reference-free comparison of generated and real token trees, scored within the correct regime, should therefore track human preference in exactly the depth range where reference-based metrics go blind.
Core Claim
The claim has two parts, and the second is useful even if the first fails.
The first part is explanatory. For a corpus of $n$ frames whose per-layer codes have effective entropy $h$ bits, trie theory places the profile transition near depth $k^* \approx \log_2 n / h$. With $n$ in the tens of millions and DAC's 10-bit codebooks partly underused, $k^*$ lands around layers 2 to 3. That is suspiciously close to the LSD plateau. If the connection is real, the plateau layer should move predictably when the effective $n$ or $h$ changes, for example across training-set sizes, content types, or codebook utilization. A generic "diminishing returns" story predicts no such movement.
The second part is a measurement tool, a regime-aware profile divergence. It compares the depth profile of generated token streams with that of real audio at matched corpus size. It weights depths by the regime they fall in, and it separates two sources of mismatch: wrong per-layer code frequencies, and wrong dependence between layers. The prediction is that the deep-regime component of this divergence carries information about human preference that LSD, ViSQOL, and Fréchet Audio Distance (FAD, a distance between embedding statistics of real and generated sets) lack.
What the theory supplies, and what it does not
The mathematical side rests on a small set of results. On the average profile of symmetric digital search trees maps five asymptotic regimes of the expected profile for unbiased binary trees. (Un)expected behavior of digital search tree profile extends this to biased sources. It shows a nearly complete tree below an entropy-dependent threshold on $k/\log n$, a phase transition at that threshold, and oscillating polynomial growth beyond it, derived by Poissonization and Mellin transforms. Average Complexity of Additive Properties for Multiway Tries shows that alphabet bias makes depth variance grow like $\log n$ instead of staying bounded. For our purposes this means skewed codebooks spread frames across depths more unevenly.
We reuse only the Poissonized form of the expected profile. It stays exact enough to compute numerically for a 1024-ary alphabet. The null model is a source that is independent across layers, with per-layer code frequencies $p_j$ fitted from real audio, so that the prefix $w = (w_1,\dots,w_k)$ has probability $P(w) = \prod_{j\le k} p_j(w_j)$. The expected number of branching nodes at depth $k$, meaning prefixes shared by at least two of $n$ frames, is then
$$\mathbb{E}[X_{n,k}] = \sum_{|w|=k}\left(1 - e^{-nP(w)} - nP(w)\,e^{-nP(w)}\right).$$
Here each term is the Poisson probability that prefix $w$ is hit at least twice.
Real codec codes are not independent across layers, and the theory above assumes memoryless sources. The file names extension to Markov sources as an open problem. We therefore do not claim the theory describes codec streams exactly. We use it as a calibrated null in two ways. It says which depths are informative for a given $n$. It also lets the observed profile be split into the part explained by per-layer frequencies and the excess caused by cross-layer dependence.
We treat the synthesis table in the research file skeptically. It assigns Knessl's five regimes one-to-one onto layers 1 through 9, which is a loose analogy, because the regimes are defined asymptotically in $n$ and not by layer index. In this proposal the regime of a layer is a computed quantity that depends on $n$ and $h$. It is never assigned by fiat.
Improved Behaviour of Tries by the "Symmetrization" of the Source shows that re-coding a biased source restores balanced trie behavior. Probabilistic behavior of asymmetric level compressed tries shows that structural compression alone cannot remove the effect of bias. Both constrain the design. Codebook skew is a property of the source that the profile must account for, not something to normalize away before measuring.
Research Program
The first step is empirical profiling. We encode MTG-Jamendo, FMA, and Common Voice with DAC at 44.1 kHz and nine layers, the same setup as the codec resynthesis study. We build prefix tries over frame code stacks at corpus sizes spanning three decades. We then check that the observed profiles show the predicted near-complete, transitional, and unique regimes, and that the transition depth scales with $\log n / h$ as the null model predicts. Where it departs, we measure the size of the cross-layer dependence excess.
The second step is the plateau test. We train the resynthesis model of Geometric Iterative Retrieval, or a smaller replica, on subsets whose sizes differ by factors of ten, and on speech versus music. For each, we record the layer at which LSD plateaus. The explanatory claim predicts that the plateau layer shifts by roughly $\log_2 10 / h$ layers per decade of data, around 0.4 layers for $h \approx 8$ bits, and differs between speech and music according to their measured entropies. The shift per decade is small, which is why the design spans three decades.
The third step is the divergence metric. For a set of generated frames, we compute its profile at matched $n$ against a real reference. The reference is available in two forms: the analytic null, and an empirical real-audio profile subsampled to the same $n$. The divergence is a regime-weighted log-ratio of profiles, reported separately for shallow, transitional, and deep regimes. The deep component is further split into frequency-driven and dependence-driven parts. The metric needs only codec encoding and prefix counting, so it runs in seconds per system. That matters because critique-based evaluators such as MuseCritic are too slow to use during training.
Baselines
The naive reference point is the full-reference metrics the codec literature already uses. These are LSD, and ViSQOL v3, a perceptual similarity metric with fine time alignment, silence thresholds, and a regression mapping onto mean opinion scores. Both are run on identical resynthesized clips. The reported numbers we anchor to come from the resynthesis paper: LSD of 10.61 for geometric retrieval versus 12.42 for a cross-entropy baseline, a 45-point human preference lead on a 100-point scale, and an LSD plateau at three layers. Beating LSD here means our deep-regime component rises monotonically across layers 4 to 9, in the same order listeners rank them, while LSD stays flat.
The reference-free distributional baseline is FAD over VGGish and CLAP embeddings, following the practice benchmarked in A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset. That study reports StyleGAN2 reaching FAD 2.22 on Looperman, and showed that distributional metrics track texture rather than rhythmic grammar. We compute FAD at the same sample counts as the profile metric, because both are biased at small $n$.
The strongest learned competitors are trained human-preference predictors. The model trained on SongEval reports a Pearson correlation of 0.912 with human averages, against 0.61 to 0.66 for Audiobox-Aesthetic. CMI-RewardBench reports 78.20% accuracy on the CMI-Pref musicality task and 73.43% on Music Arena. MuseCritic reports 71.35% pairwise accuracy on Music Arena. We do not expect a training-free counting statistic to beat 0.912 on song aesthetics, so beating these models outright is not the bar. Instead there are two tests. First, does adding our deep-regime component to the learned predictor raise held-out correlation, or pairwise accuracy on the codec layer-ablation set that these models were not built for? Second, on that ablation set, does our metric alone rank layers as well as they do, at a small fraction of the cost? For profile-based audio metrics specifically, no reported result was found on Lacuna.
Three ablations isolate the key ingredient. The first uses marginal per-layer codebook entropy and utilization, which tests whether cross-layer structure matters at all. The second uses an empirical prefix-count comparison without regime weighting, which tests whether the theory adds anything beyond counting. The third uses the analytic null versus the subsampled empirical reference, which tests whether the memoryless theory is adequate as a reference. All variants use the same frames, the same $n$, and the same codec.
Why not just…?
Why not just measure per-layer codebook entropy of generated tokens? This is the simplest competitor, and it may capture most of the signal if deep-layer failures are mainly generators collapsing onto frequent codes. It cannot, however, detect generators that reproduce the right code frequencies at each layer while getting the pairing between layers wrong. That pairing is what distinguishes an averaged, mean-seeking residual from a coherent one, the failure mode the resynthesis paper attributes to one-step regression. The frequency-versus-dependence split of the deep component tests this directly. If the dependence part adds nothing to human-preference prediction beyond marginal entropy, we will report that marginal entropy suffices and narrow the claim to the explanatory plateau result.
Why not just train a better learned evaluator? Learned evaluators are strong on the distributions they were trained on. They are also slow, entangled, and prone to compressed score ranges. CMI-RewardBench reports an SRCC around 0.85 between its musicality and alignment heads, and MuseCritic documents direct score prediction clustering around 3.5 to 4.0. None of them says which depth of the representation is wrong. Our statistic is meant as a cheap, interpretable complement that can run inside a training loop. The augmentation experiment settles whether it adds information or merely duplicates what learned models already capture.
Why not just calibrate full-reference metrics, as ViSQOL v3 does? ViSQOL v3's alignment and silence fixes address transmission artifacts. The file notes the metric still misjudges generative codecs that synthesize plausible detail absent from the reference. Our hypothesis gives the reason: past the transition depth, the reference is one sample from a high-entropy conditional distribution, so no alignment can make matching it the right target. Running ViSQOL on the layer ablation tests this. If calibrated ViSQOL tracks listeners through layer 9, the hypothesis is wrong.
Why not skip the asymptotics and compare empirical tries directly? This is the real threat to the theoretical half. At matched $n$, a subsampled real-audio trie may be as good a reference as the analytic null. If so, the metric keeps the empirical reference. The theory's remaining contribution is then threefold: the regime boundaries that decide which depths to weight, the dependence decomposition, and the falsifiable prediction that the plateau layer moves with $\log n / h$. That prediction cannot come from empirical counting alone.
Proposed Experiments and Decisive Checks
The first experiment is a new double-blind listening test on the resynthesis layer ablation. It covers layers 1 through 9 on MTG-Jamendo, FMA, and Common Voice clips, with enough raters to resolve adjacent-layer preferences. We then correlate LSD, ViSQOL, FAD, marginal entropy, and each profile component with those preferences. The key test is layers 4 to 9, where LSD is flat.
The second experiment is the plateau-shift study across data scale and content type. We regard it as the decisive check on the explanatory claim. A plateau layer that ignores $n$ and $h$ falsifies it.
The third experiment transfers the metric to generated music. We encode SongEval's 2,399 songs and MusicEval's 2,748 clips with DAC and correlate the profile components with the expert ratings, especially the vocal naturalness and musicality dimensions. We then test whether adding the deep-regime component raises the correlation of a SongEval-trained predictor on held-out systems. Held-out systems are the relevant split, because the file notes that learned evaluators generalize unevenly.
The project succeeds if the deep-regime component ranks layers as listeners do, where LSD cannot, and the dependence part carries signal beyond marginal entropy. It partially succeeds if only the plateau-shift prediction holds. It fails cleanly if marginal entropy or calibrated ViSQOL already does the job.
References
- Geometric Iterative Retrieval for Neural Audio Codec Resynthesis
- On the average profile of symmetric digital search trees
- (Un)expected behavior of digital search tree profile
- Average Complexity of Additive Properties for Multiway Tries: A Unified Approach (Extended Abstract)
- Improved Behaviour of Tries by the "Symmetrization" of the Source
- Probabilistic behavior of asymmetric level compressed tries
- MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques
- ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric
- A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction