Large audio classifiers are accurate but opaque. The Audio Spectrogram Transformer reaches 95.6% on the ESC-50 environmental-sound benchmark, while the best listenable, self-explaining model in our material reaches 82.5%. That gap is usually described as an unavoidable accuracy-interpretability trade-off, but nobody measures what fills it. We propose to route a frozen audio transformer's features through a stack of discrete codebooks, which are small learned dictionaries of vectors where each input frame is replaced by its nearest entry. The first codebook captures coarse sound patterns. Each later codebook quantizes whatever the previous one missed. A sparse linear classifier then reads only which codes fired, and nothing continuous.
Each code can be played back as the training audio segments assigned to it. Adding one codebook stage at a time therefore produces a ladder of models. Each rung is fully auditable, and the accuracy gained at each rung is paid for by a specific, listenable set of new concepts. The trade-off stops being a single slogan and becomes a curve, with a measurable remainder at the top: the continuous residue that no code explains.
Core Claim
Our hypothesis has two parts. First, a head that reads only hard code assignments cannot hide information in small continuous activations, so it is interpretable by construction. Second, stacking two or three residual codebook stages recovers most of the accuracy that a single hard codebook loses, while keeping the stability that hard quantization gives. If this holds, it settles a live disagreement in the literature, described below, and gives audio the first prototype-style classifier whose explanations are sounds rather than heatmaps.
The formal mechanism is residual vector quantization. Let $h$ be a frame feature from the frozen encoder, and set $r_0 = h$. At stage $s$, pick the nearest entry $q_s$ of codebook $E_s$ to $r_{s-1}$, then pass on $r_s = r_{s-1} - q_s$. The clip-level input to the head is a vector $a_s$ counting how often each code of $E_s$ was selected across the clip's frames. The prediction is
$$\hat{y} = \sum_{s=1}^{S} W_s a_s,$$
where each $W_s$ is a sparse, non-negative weight matrix and $S$ is the number of stages kept. We call $\text{Acc}(S)$ the accuracy ladder. The final residue $r_S$ is never shown to the head. Instead, a separate probe on $r_S$ measures how much class information the codes failed to capture.
Where This Sits
Three vision papers motivate the design and disagree about it. ProtoQuant snaps frozen-backbone patch features to a single hard codebook and classifies with a sparse linear layer over max-pooled cosine similarities. It reports much more stable prototypes than continuous ones: a Prototype Activation Change of 0.07 versus up to 23.70 for earlier methods. Escaping Low-Rank Traps argues the opposite. In its view, hard quantization destroys information, so its codebook only pulls continuous features toward anchors while the continuous features are passed forward. We take a third position between these two: hard codes, but several residual stages of them.
We also depart from ProtoQuant in one specific way. Its head reads continuous similarity magnitudes, so it can still smuggle information through how similar a patch is. Our head reads counts only. We test directly whether that difference matters.
Codebook Features supplies the key supporting evidence: discrete codes in Transformers remove information smuggling and raise monosemantic precision from 70.5% to 97.1% on a synthetic state machine. That work concerns language-model internals, not perception classifiers with grounded, playable prototypes. Concept Bottleneck Models, which predict human-labelled concepts first and the label second, identify leakage through continuous concept activations as the core failure of bottleneck designs. We inherit their test-time intervention idea but use unsupervised codes, because audio datasets lack concept annotations.
On the audio side, L2I shows that spectrogram saliency maps fail human listeners. It explains predictions with listenable components from a non-negative matrix factorization dictionary, which splits spectrograms into additive parts. Its dictionary is linear and shallow, which likely explains its accuracy ceiling. Two further papers use the leftover after quantization. Explainable Disentanglement on Discrete Speech Representations uses the quantization residue as a noise channel, supervised with noise labels. We instead quantize the residue into further concept stages and treat whatever remains as unexplained. Bringing Interpretability to Neural Audio Codecs shows post hoc that residual-quantized codec tokens entangle content, speaker and pitch. That is a warning that later residual stages may be less clean than early ones, and we measure this rather than assume it away.
Research Program
Model. We freeze a pretrained AST (AST) and extract features from one layer; the choice of layer is itself swept. We train $S \le 4$ residual codebooks with standard commitment losses and dead-code restarts, and fit the sparse non-negative heads. We train stages in two ways: jointly, and greedily one stage at a time. Greedy training keeps earlier rungs fixed, so each rung of the ladder is exactly a sub-model of the next.
Grounding. Every code is grounded by retrieving its most frequently assigned training segments. These segments are what a listener hears as the explanation for that code.
Measurements. For every rung we measure three things:
- Accuracy, giving the ladder $\text{Acc}(S)$.
- Stability: an audio analogue of Prototype Activation Change, computed under time shifts, gain changes and added background noise at controlled signal-to-noise ratios.
- Remaining information: a probe on $r_S$ that predicts the class. Its accuracy above chance quantifies the part of the trade-off still unexplained.
Interpretability tests.
- Smuggling test. On ProtoQuant-style similarity heads, we perturb the similarity magnitudes while preserving code assignments. Accuracy that moves under this perturbation reveals reliance on continuous magnitude.
- Intervention test. We silence the codes most responsible for a prediction and measure how often the prediction flips.
- Blind listening study. Raters judge whether a code's retrieved segments share a nameable sound. We report this separately per stage to detect whether deeper rungs become uninterpretable.
Baselines
Naive reference. A linear probe on mean-pooled frozen AST features. It uses the same encoder and data, with no interpretability constraint, and it is the accuracy ceiling our heads can reach without fine-tuning. No reported result was found on Lacuna for frozen-AST linear probes on ESC-50, so we run it ourselves. The fully fine-tuned AST (95.6% on ESC-50 and 0.485 mAP on AudioSet, from AST) is reported as an outer ceiling only, since it uses far more trainable capacity.
Strongest interpretable audio baseline. L2I, which reports 82.5% on ESC-50 and 0.831 F1 on OpenMIC-2018. Beating it means exceeding those numbers with a head that reads only code counts, on the same splits.
Strongest structural competitor. ProtoQuant ported to audio. It is the most direct existing recipe: a single hard codebook with a cosine-similarity head, on the same frozen AST features, with the same total number of codes and the same pruning to top-$K$ codes per class. Its reported vision results (87.6% on CUB-200 with ConvNeXt-Tiny, versus 84.3% for PIP-Net) anchor the vision check below.
Soft-regularization competitor. An implicit-quantization head in the spirit of Escaping Low-Rank Traps, where anchors pull the features but continuous features reach the classifier.
Fairness. All methods share the encoder layer, data splits, codebook size budget, number of head nonzeros, and an equal hyperparameter-search budget.
Ablations. Two ablations isolate our key ingredients. The first replaces residual stages with one codebook containing the same total number of codes. The second swaps count inputs for similarity inputs.
What counts as success. Our method wins if its best rung comes within a few points of the linear probe while matching the audio ProtoQuant port on stability, and if the count head passes the smuggling test where the similarity head fails.
Why not just…?
Why not just use one hard codebook, as ProtoQuant does? This is the real threat. ProtoQuant loses little accuracy in vision, so a single codebook on AST features might already approach the linear probe. The equal-code-count ablation settles it. If single-stage $\text{Acc}(1)$ is within about one point of the probe, we report that residual stages are unnecessary for accuracy. The contribution then reduces to the count-versus-similarity smuggling result and the residue audit, which remain informative either way.
Why not just regularize softly and keep continuous features? Soft anchors protect representation rank: Escaping Low-Rank Traps reports that baseline concept-model feature rank falls from 196 to 70 while theirs stays high. However, the classifier still sees continuous features, which is exactly the channel that Codebook Features and Concept Bottleneck Models identify as a route for hidden information. Our stability and smuggling measurements compare the two approaches head to head. If soft regularization matches us on both, the case for hard codes weakens accordingly.
Why not just explain AST post hoc? L2I reports that listeners found listenable decompositions more helpful than audio LIME, and spectrogram heatmaps do not match how people hear sound. We also include a heatmap condition in the listening study, so this claim is re-tested rather than assumed.
Proposed Experiments
The primary benchmarks are ESC-50 (five-fold), OpenMIC-2018 and SONYC-UST, with AudioSet as the large multi-label stress test. Speech Commands V2 serves as a probe of a predicted failure. High-frequency fricatives resemble noise and leak into the residue in the ASR disentanglement work, so we expect keywords separated only by fricatives to be recovered only at deeper rungs.
As a bridge back to the vision debate, we run the same ladder on CUB-200 with ConvNeXt-Tiny against ProtoQuant's reported 87.6%.
Three outcomes are decisive:
- Shape of the ladder. Does $\text{Acc}(S)$ rise meaningfully beyond $S=1$ at an equal code count?
- Leakage. Does the ProtoQuant-style similarity head measurably rely on continuous magnitudes that the count head cannot use?
- Interpretability of deep rungs. Do stage-two and stage-three codes remain nameable to listeners? If they do not, deeper rungs buy accuracy with codes that only appear interpretable, and we will report that as the measured price of the trade-off.
References
- ProtoQuant: Quantization of Prototypical Parts For General and Fine-Grained Image Classification
- Escaping Low-Rank Traps: Interpretable Visual Concept Learning via Implicit Vector Quantization
- Codebook Features: Sparse and Discrete Interpretability for Neural Networks
- Concept Bottleneck Models
- Tackling Interpretability in Audio Classification Networks With Non-negative Matrix Factorization
- Explainable Disentanglement on Discrete Speech Representations for Noise-Robust ASR
- Bringing Interpretability to Neural Audio Codecs
- AST: Audio Spectrogram Transformer