Joint Analysis of Latent Dimensionality and Frame Rate in Continuous Audio Encoders

Kyudan Jung, Sehyun Lee, Son-ha Jo, Jaegul Choo, Sanghyuk ShoiarXiv 2026 · arXiv

TL;DR

Optimizing neural audio encoders solely for perfect sound reconstruction can actually hurt their performance on downstream tasks like speech recognition and question answering. This paper reveals a critical "width–rate interaction": as you lower the frame rate (temporal compression), the encoder requires a larger latent width (feature capacity) to effectively organize and retain task-relevant information. Crucially, the authors show that how a bottleneck constrains a model during training matters far more than simply pruning excess dimensions after the fact.

What Problem Does This Paper Address?

Neural audio codecs are increasingly used to convert continuous audio waveforms into discrete or continuous latent representations for downstream speech language models (like audio LLMs). These encoders compress audio along two primary axes: latent width ($d$, the dimensionality of each frame vector) and frame rate ($f$, how many vectors are produced per second).

Traditionally, researchers select these settings based on reconstruction metrics—how well a decoder can recreate the original audio waveform from the latent code. Unsurprisingly, making the latents wider and more frequent yields the highest audio fidelity. However, it remains unclear how these two axes jointly impact downstream semantic tasks like Automatic Speech Recognition (ASR) or Spoken Question Answering (SQA). Does a tighter compression bottleneck force the encoder to learn better, more organized semantic features, or does it simply discard crucial information?

Methods

The authors systematically explore this design space by training sixteen different audio encoders covering all combinations of:

  • Latent Widths ($d$): 128, 256, 512, and 1024 dimensions.
  • Frame Rates ($f$): 3.125, 6.25, 12.5, and 25 Hz.

Architecture & Training

They use a Descript Audio Codec (DAC-VAE) backbone. To alter the frame rate natively, they introduce learned stride-2 convolutional downsamplers before the bottleneck and corresponding upsamplers after it. Every encoder configuration is trained from scratch using the exact same multi-scale spectrogram and adversarial loss functions.

Downstream Evaluation

To test the utility of these representations, the authors freeze the trained encoders and connect them to a frozen Qwen3.5-4B language model via a trainable 4-layer Transformer adapter. They evaluate this setup on ASR and SQA. For speaker identification, they apply a simple linear probe over pooled frame statistics.

Principal Component Analysis (PCA) Interventions

To understand if wider models simply contain redundant information, the authors perform post-hoc PCA on the frozen representations, zeroing out the trailing 50% of variance components, and then feed these pruned representations back into the downstream models and decoders without retraining.

Datasets

  • Encoder Training: A massive mixed dataset containing speech, general audio, and music from LibriSpeech, WavCaps, and MAESTRO.
  • ASR & SQA Training: 15.5 million utterances pooled from MLS English, GigaSpeech, LibriTTS-R, English VoxPopuli, and InstructS2S-200K.
  • Evaluation Benchmarks:
    • ASR: LibriSpeech test-clean (measured via Word Error Rate, WER).
  • SQA: mistralai/triviaqa_speech (measured via Cover Exact Match, Cover-EM).
  • Speaker ID: VoxCeleb1 and LibriSpeech train-clean-100.

Key Insights and Ideas

  • The Reconstruction vs. Downstream Mismatch: Perfect audio reconstruction does not equal optimal downstream performance. While the highest frame rate (25 Hz) and largest width (1024d) yield the best acoustic fidelity, downstream tasks consistently peak at a lower, more compressed frame rate (12.5 Hz).
  • The Width–Rate Interaction: Sparser frames need more per-frame capacity. At higher frame rates (12.5–25 Hz), downstream models achieve the best performance using moderate widths (256d–512d). However, as the frame rate drops to 3.125 Hz, the optimal width shifts decisively to the largest capacity (1024d).
  • The "Training Bottleneck" Effect: You cannot replicate the benefits of a narrow bottleneck by simply pruning a wide model post-hoc. When the authors took a 1024d model and dropped its bottom 50% PCA dimensions, its performance barely changed. Yet, this pruned model still performed worse than a model that was natively trained at 512d or 256d. The physical constraint during training forces the network to organize information differently.

Key Results

Reconstruction vs. Downstream Performance

As shown in Table 1, acoustic reconstruction metrics (PESQ and SI-SDR) scale monotonically with width and frame rate, peaking at 25 Hz / 1024d:

Table 1. Speech reconstruction results across all sixteen en- coders. PESQ and SI-SDR are higher when better. Bold indi- cates the best result in each row.

However, Table 2 reveals that downstream tasks tell a completely different story. ASR and SQA metrics peak at 12.5 Hz, with ASR favoring 256d (3.77% WER) and SQA favoring 512d (34.58% Cover-EM). At the lowest frame rate (3.125 Hz), the preference shifts heavily toward 1024d:

Table 2. Downstream results for sixteen configurations. Bold marks the best result per metric in each row. Paired-bootstrap 95% CIs have ≈0.2-pp half-widths for WER differences and include zero for SQA differences < 1 pp; test-set sampling only.

PCA Pruning Sensitivity

When applying the top-half PCA ablation (Table 3), the 1024d models show incredible robustness—their ASR performance remains virtually unchanged (and sometimes slightly improves) despite losing half their dimensions. Conversely, narrower models (like 512d at 12.5 Hz) collapse dramatically when cut in half, with WER shooting up from 3.83% to 30.48%:

Table 3. Fixed-weight top-half PCA ablation. “Half” re- ports performance after retaining the leading half of the PCA components. ∆denotes the relative perfor- mance change: 100(WERFull/WERHalf −1) for ASR and 100(AccHalf/AccFull −1) for speaker identification. Darker red indicates a larger absolute change.

Interestingly, Table 4 demonstrates that the audio decoder is highly tolerant to this PCA pruning across the board, further proving that what is "reconstructible" by a codec decoder is not the same as what is "accessible" to a downstream language model.

Table 4. Codec reconstruction after the ASR top-half PCA intervention. “Half” reports reconstruction performance after retain- ing the leading half of the PCA components. ∆denotes the difference between Half and the corresponding full-representation result reported in Table 1. For each reconstruction metric, blue intensity is normalized across all sixteen configurations, with darker blue indicating better performance. Darker red indicates a larger absolute degradation.

Limitations and Future Work

While this work clearly demonstrates the width–rate interaction, the authors note a few limitations:

  1. PCA Alignment: PCA ranks features purely by variance, not by task relevance. A supervised dimensionality reduction technique might reveal different sensitivity profiles.
  2. Proving the Mechanism: While the results strongly support the "information bottleneck" hypothesis (that compression pressure forces better semantic organization), fully proving this exact mechanism will require more tightly controlled training experiments.
  3. Acoustic Metric Limitations: Standard reconstruction metrics use cropped inputs and aggregate averages, which might fail to capture the loss of fine-grained linguistic cues that cause downstream models to fail.

Summary updated

Abstract

Continuous audio encoders compress audio along feature and time axes through latent width and frame rate, but their joint effect on downstream performance remains unclear. We train sixteen encoders spanning four widths and four frame rates, with downstream adapters and probes, using matched training protocols. Despite generally improved reconstruction at larger widths, automatic speech recognition (ASR) and spoken question answering (SQA) favor moderate widths at higher rates, with the best observed widths shifting toward larger values under stronger temporal compression. Frozen-model PCA interventions reveal distinct reconstruction and recognition sensitivities: removing the trailing half of the components substantially degrades ASR in high-rate 512-dimensional encoders with comparatively small reconstruction penalties, whereas 1024-dimensional encoders largely preserve both. Yet the projected 1024-dimensional model underperforms unmodified narrower models on ASR at 12.5Hz. These findings identify a width--rate interaction in downstream utility and suggest that how representations are organized during training matters beyond reconstruction fidelity and compressibility.
BibTeX
@misc{jung2026joint,
  author = {Kyudan Jung and Sehyun Lee and Son-ha Jo and Jaegul Choo and Sanghyuk Shoi},
  title = {{Joint Analysis of Latent Dimensionality and Frame Rate in Continuous Audio Encoders}},
  year = {2026},
  eprint = {2609.29780},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.29780},
}
Figure