Audio language models first turn sound into integers using a neural codec, a network that compresses a waveform into a short sequence of discrete codebook indices. A single codec now has to serve speech, music, and environmental sound. Each clip therefore arrives with a hidden question the system must answer: what kind of sound is this, and what are its particular settings, such as the speaker, the instrument, or the room? Information theory gives a precise price for answering that question from the data itself. When a compressor does not know which member of a $d$-parameter family of sources it faces, it pays an extra cost of about $\frac{d}{2}\log n$ bits over a block of $n$ symbols. This is the "third-order" model-cost term. It grows only logarithmically, so it is invisible to first-order (entropy) and second-order (dispersion) analyses.
We propose to measure this price directly on codec token streams and to use it to make tokenizer design choices. The core observation is that codec tokens are already discrete. Losslessly coding them with an autoregressive token model is exactly the setting in which the sharp third-order results of Kosut and Sankar and Iri and Kosut hold. The codec has done the continuous-to-discrete step, so the open problem of lossy third-order theory can be sidestepped. The effective dimension $d$ of cross-domain uncertainty can then be read off from how a token model's loss falls with position in a clip. That number predicts which tokenizer wins at which clip length.
Core Claim
Suppose a token model is reset at the start of each clip and trained on a mixture of domains. For a parametric source, its expected excess loss at position $t$, relative to a model that already knows the clip's source, should decay like
$$\ell(t) - h \approx \frac{d_{\mathrm{eff}}}{2t},$$
where $\ell(t)$ is the average per-token loss in nats at position $t$, $h$ is the long-context loss floor, and $d_{\mathrm{eff}}$ is the effective number of continuous source parameters the model must infer in context. Summing this over $t \le n$ gives the classical $\frac{d}{2}\ln n$ regret of Bayes-mixture universal coding.
This yields three testable claims. First, the decay exponent should be close to 1 per domain, which is the signature of a finite-dimensional source. An exponent well below 1 would indicate a nonparametric regime, and we would report that outcome. Second, the gap between a mixed-domain model and a domain-oracle model should grow logarithmically with clip length if within-domain continuous variation dominates. If instead the domain is essentially a discrete three-way label, the gap should saturate at a constant near $\ln 3$. Third, the measured $d_{\mathrm{eff}}$ should be large for the first residual codebook level and near zero for later levels. This turns the qualitative finding of Analysis of Semantic and Acoustic Token Variability into a bit count.
The design consequence follows from the alphabet-size dependence in Kosut and Sankar. Their third-order coefficient for memoryless sources grows with support size, as $\frac{|\mathcal{X}_p|-1}{2}$ for prefix codes. A large single codebook like WavTokenizer's 16,384 entries may therefore carry a larger model cost than hierarchical residual quantization. The penalty is amortized as $(\log n)/n$, so the comparison should flip with clip length. We predict a measurable crossover clip duration below which hierarchical tokenizers cost fewer total bits at matched quality, and above which the large single codebook wins.
Relation to Prior Work
Three groups of papers play different roles here.
The model-cost theory supplies the method. Kosut and Sankar establish exact third-order coefficients for universal lossless coding of memoryless sources. Iri and Kosut show that for $d$-dimensional exponential families the optimal third-order rate is governed by the true dimension $d$. They reach it by partitioning the continuous space of sufficient statistics into cuboids. Both papers state that their results are lossless only. We do not try to close the lossy gap. Instead, we apply their lossless theory where it is valid, to the discrete token stream that sits after the codec.
The continuous-to-discrete literature constrains how we interpret frame rate and quantization resolution. Sakai et al. slice continuous time into vanishing bins and recover a $\log T$ third-order law in duration. By analogy, the model cost of a clip of duration $T$ at frame rate $f$ should behave like $\frac{d}{2}\log(fT)$. Frame rate should therefore enter only additively in the log, which we can check across tokenizers running at 5, 40, and 75 tokens per second. Fong and Tan quantize continuous inputs at a resolution that grows polynomially in blocklength, fine enough to preserve information and coarse enough to keep the alphabet manageable. We reuse this idea when we test a quantized per-clip parameter header.
The audio work supplies the evidence for the gap. Analysis of Semantic and Acoustic Token Variability finds that the first codebook level carries most of the domain-identifying variation, using Zipf, n-gram perplexity, and TF-IDF statistics. It gives no rate interpretation, and its n-gram models cannot reveal how in-context adaptation cost scales with length. SUNAC conditions a codec on discrete source prompts such as Speech, Music, or SFX, but it does not ask what that conditioning saves in bits. WavTokenizer argues for a single large codebook and is evaluated almost entirely on clean speech. Our crossover prediction tests exactly the regime it leaves open. Codec-SUPERB supplies the matched-quality yardstick.
Research Program
We will tokenize matched-duration corpora in three domains with frozen public tokenizers. The tokenizers are EnCodec and DAC, which use residual vector quantization, and WavTokenizer at 40 and 75 tokens per second. The corpora follow the token variability study: LibriTTS for speech, MUSDB18 and MTG-Jamendo for music, and AudioSet clips without speech or music labels for sound.
For each tokenizer we will train small autoregressive transformers in two configurations: one on the domain mixture, and one per domain as the oracle. We will also train a separate model for each residual level, conditioned on the coarser levels. Context is reset at clip boundaries.
From held-out clips we will estimate the per-position loss curve $\ell(t)$. We will fit the decay exponent and the coefficient $d_{\mathrm{eff}}$, with the floor $h$ estimated from the tail of long clips. Bootstrap confidence intervals will be computed over clips. We will also record the quantiles of the total code length to check the fixed-error-probability form, in which Iri and Kosut predict a $(d-1)/2$ coefficient alongside a square-root dispersion term.
A second analysis decomposes the cost by where the model is uncertain. We will compare the mixed-domain model with and without a ground-truth domain label against the per-domain oracle. The label-conditioned gap isolates within-domain continuous variation. The label's own contribution should be bounded by a constant.
The third component is an intervention. We will add a per-clip header: a low-dimensional continuous summary of the clip, learned jointly with the token model and quantized uniformly. Following the Laplace-method argument behind cuboid partitions, the cell width scales as $c\,n^{-1/2}$ per dimension. The header is transmitted once, and the token model is conditioned on it.
Theory says a two-part code of this kind should match the Bayes mixture to within a constant, not beat it. So the header is a test of the theory, not a claimed improvement in lossless rate. Its practical value is that it moves the identification cost to the start of the clip, which matters for streaming and for non-autoregressive decoders. Its resolution law is a falsifiable prediction: the best cell width should shrink as $n^{-1/2}$, and a fixed-resolution header should lose ground as clips lengthen.
Baselines
The naive reference point is the codec's nominal bitrate with no entropy coding, for example 0.9 kbps for WavTokenizer at 75 tokens per second. Every measured saving is expressed against this. It shows how much of the fixed-rate budget is redundancy that a universal coder recovers.
The strongest existing competitor is the mixed-domain token model itself with long context. A well-trained transformer approximates the Bayes mixture, which is the optimal universal code. The header and any label conditioning must therefore be compared against it at an identical parameter count, identical training tokens, and the same tuning budget, with learning rate and width swept equally for all variants. No reported result found on Lacuna for per-token code length of codec token streams under any model, so these numbers will be produced here.
The SUNAC-style discrete domain label is the baseline that isolates our key ingredient, continuous within-domain conditioning. SUNAC conditions on Speech, Music, or SFX prompts, and the file reports its efficiency (9.5 GMACs per source versus 51.5 for cascaded TUSS plus DAC) but no bitrate saving. We implement label conditioning inside our token models, trained identically. The continuous header "beats" it if the label-conditioned gap to the oracle still grows as $\log n$, while the header gap stays bounded with confidence intervals excluding zero difference at clip lengths of 10 seconds and above.
For the tokenizer crossover, the strongest reported quality anchors are WavTokenizer's UTMOS of 4.05 at 0.9 kbps on LibriTTS (WavTokenizer) and AFACodec's Whisper WER of 3.07% and CLAP event accuracy of 88% at 7–8 kbps on Codec-SUPERB. Tokenizer comparisons are made only at matched Codec-SUPERB quality, using the same downstream expert networks, so that bit savings are not bought with fidelity.
A per-level ablation, which models only codebook level one with all others held fixed, tests whether the model cost really concentrates in the first residual level.
Why not just…?
Why not just condition on a discrete domain label, as SUNAC does? This is a real threat. If cross-domain uncertainty is essentially three-way, a label costing $\log_2 3$ bits per clip captures all of it, and the third-order view adds nothing beyond a constant. The label-conditioned versus oracle gap settles this directly. A saturating gap means the label suffices and our continuous-header proposal fails, which would itself be a useful negative result. A gap growing as $\log n$ means speaker, instrument, and room parameters dominate.
Why not just use a bigger model with longer context and let it adapt in context? In lossless terms this is the right answer. The Bayes mixture is optimal, and we do not claim the header beats it. What longer context cannot do is tell you how much adaptation costs or how that cost depends on tokenizer and clip length. The loss-versus-position fit provides that, and it is what drives the crossover prediction. If the decay exponent is well below 1, the parametric picture is wrong and the coefficient $d_{\mathrm{eff}}$ is meaningless. We will report that outcome rather than force a fit.
Why not just report per-domain perplexity, or use a second-order dispersion approximation? The token variability study already shows perplexity is similar across domains. That is exactly why the domain-identification cost cannot be seen in it. Average perplexity mixes the floor $h$ with the decaying excess. Second-order dispersion describes fluctuations around a known source and, as the universal-coding results show, contains no term for parameter learning. Only the position-resolved curve separates the two.
Proposed Experiments and Decisive Checks
The first check is the exponent. Per-domain loss curves on EnCodec, DAC, and WavTokenizer tokens should decay with exponent near 1, and the fitted $d_{\mathrm{eff}}$ should be stable across clip-length ranges.
The second check is the frame-rate law. Comparing WavTokenizer at 40 and 75 tokens per second on identical audio, total model cost per clip should differ by about $\frac{d}{2}\ln(75/40)$ nats, not in proportion to token count. A proportional difference would reject the Sakai-style continuous-time reading.
The third check is the crossover. On mixed-domain clips of 1 to 30 seconds, including Divide and Remaster mixtures for dense content, we measure total entropy-coded bits for each tokenizer at matched Codec-SUPERB quality. We predict that the crossover duration can be computed in advance from the fitted $d_{\mathrm{eff}}$ and floor values of each tokenizer. The prediction counts as confirmed if the computed crossover falls within the bootstrap interval of the observed one.
The fourth check is the header resolution sweep. The optimal cell width should scale as $n^{-1/2}$. A fixed-width header should fall behind the mixture code by an amount growing with $n$, while the correctly scaled header stays within a constant.
If all four checks hold, codec designers gain a cheap, length-aware number for choosing between large single codebooks and residual hierarchies. If they fail, we learn that audio tokens are not well described as a finite-dimensional source.
References
- Asymptotics and Non-Asymptotics for Universal Fixed-to-Variable Source Coding
- Fine Asymptotics for Universal One-to-One Compression of Parametric Sources
- Analysis of Semantic and Acoustic Token Variability Across Speech, Music, and Audio Domains
- WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling
- Second-Order Asymptotics of the Continuous-Time Poisson Channel
- A proof of the strong converse theorem for Gaussian multiple access channels
- SUNAC: Source-aware Unified Neural Audio Codec
- Codec-Superb @ SLT 2024: A Lightweight Benchmark For Neural Audio Codec Models