TL;DR
Q2D2 (Two-Dimensional Quantization) introduces a geometry-aware approach to neural audio coding by grouping latent channels into pairs and quantizing them onto structured 2D lattices (such as rhombic or hexagonal grids). By merging the stability and guaranteed codebook utilization of Finite Scalar Quantization (FSQ) with the expressive inter-feature coupling of vector quantization, Q2D2 achieves state-of-the-art speech reconstruction at ultra-low bitrates and token counts (e.g., 53 to 166 tokens/sec) using only a single quantizer.
What Problem Does This Paper Address?
Discrete audio codecs convert continuous audio into compact sequences of discrete tokens, acting as the foundation for modern speech and audio language models. Existing codecs typically rely on one of two extremes:
- Vector Quantization (VQ / RVQ): Learns codebook vectors in higher-dimensional space. While expressive, it suffers from severe codebook underutilization (codebook collapse) and optimization instability, demanding fragile stabilization tricks like commitment losses, EMA updates, and periodic codebook re-initialization.
- Finite Scalar Quantization (FSQ): Fixes codebook points onto an implicit 1D grid per channel. While FSQ completely prevents codebook collapse and eliminates learned codebook parameters, quantizing each dimension in total isolation fails to capture correlations across feature dimensions.
The authors ask: Can we retain the simplicity, stability, and high utilization of fixed-grid quantization while capturing cross-channel correlations?
Methods
Built on top of the single-quantizer WavTokenizer framework, Q2D2 replaces standard quantization modules with fixed 2D geometric tilings:
- Channel Projection & Bounding: The encoder's latent representation is projected to an even dimension $d$ (typically $d=6$), followed by a $\tanh$ nonlinearity bounding features to $[-1, 1]$. Channels are rescaled by dimension-specific levels $l_i / 2$.
- Pairwise Grouping: Features are reshaped into $P = d/2$ two-dimensional coordinate pairs $z''_j = (z'_{2j-1}, z'_{2j})$.
- 2D Grid Snapping: Each 2D pair is snapped via nearest-neighbor assignment to a fixed analytic 2D lattice $G_j \subset \mathbb{R}^2$. The authors investigate three distinct tiling schemes:
- Rectangular grid: Uniform cartesian coordinate grid.
- Hexagonal grid: Alternating rows shifted along the x-axis by $\pm dx/4$ with row spacing $dy = \frac{\sqrt{3}}{2} dx$.
- Rhombic grid: A rectangular lattice overlaid with shifted midpoints $(dx/2, dy/2)$, yielding dense oblique packing.
- Implicit Codebook: Like FSQ, codebook vectors are not learned parameters; the codebook is implicitly defined by the Cartesian product of the 2D grids ($\prod L_j$). Gradients pass straight through via the Straight-Through Estimator (STE).
Latent x ──► Linear + tanh ──► Rescale & Pair ──► Snap to 2D Grid (e.g., Rhombic) ──► Out-Projection
▲
(Fixed 2D Tiling, STE)
Datasets
- Large-Scale Training:
- 8,000 hours general audio setup: LibriTTS, VCTK, CommonVoice (3,000h) for speech; AudioSet (2,000h) for audio; Jamendo and MUSDB18 for music.
- 150,000 hours multilingual speech setup: Emilia (En/Zh/De/Fr/Ja/Ko) and MLS (En/Fr/De/Nl/Es/It/Pt/Pl).
- Ablations & Baselines: Evaluated on LibriTTS (
test-clean,test-other), LJSpeech, and LibriSpeech (test-clean). - Semantic Evaluation: Evaluated on the speech subsets of the ARCH benchmark (RAVDESS, Audio-MNIST, SLURP, EMOVO).
Key Insights and Ideas
- Rhombic Grids Provide Optimal Packing: In direct grid comparisons, rhombic tiling consistently outperformed rectangular and hexagonal tilings on PESQ and STOI. Its oblique basis provides isotropic, space-filling coverage with minimal directional bias, better matching neural latent distributions than rigid rectangular partitions.
- Quantization Induces Pairwise Structure: Computing mutual information (MI) between coordinate pairs revealed that pre-quantization MI was near zero (orthogonal representations from the linear layer). Post-quantization MI increased sharply, confirming the 2D grid forces meaningful geometric coupling between paired dimensions.
- Sweet Spot for Dimensionality: Ablating dimension size at a fixed 1 kbps showed that $d=6$ ($[7,7,7,7,7,7]$) outperformed both $d=4$ (4 dimensions, 19 levels each) and $d=8$ (8 dimensions, 3 to 5 levels each). Too few dimensions restrict feature diversity, while too many dilute resolution.
Key Results
- Outperforming Multi-Quantizer Baselines: On LibriTTS clean speech, a single-quantizer Q2D2 at 3.3 kbps (166 tokens/s) achieved a UTMOS of 4.06, surpassing DAC at 9.0 kbps (900 tokens/s, 9 quantizers) which scored 3.91.
- Direct Comparison with FSQ and VQ: At matched 1 kbps (75 tokens/s), Q2D2 achieved 4.05 UTMOS / 2.51 PESQ / 0.92 STOI, outperforming both FSQ (3.99 UTMOS / 2.39 PESQ) and WavTokenizer VQ (3.97 UTMOS / 2.47 PESQ).
- Near-Perfect Codebook Utilization: Q2D2 achieved 92.2% to 100% codebook utilization across various bitrates without any auxiliary commitment losses, entropy regularization, or codebook re-seeding.
- Superior Semantic Retention: On the ARCH benchmark, a 53-token/s Q2D2 model beat 100–900 token/s DAC and EnCodec models in downstream classification accuracy across multiple speech tasks.
- Real-Time Efficiency: Operates at a real-time factor (RTF) of 0.0039 on an NVIDIA RTX 6000 GPU.
Limitations and Future Work
- Dimension Pairing Limits: The scheme currently operates strictly on 2D pairs; exploring 3D lattices, simplex tilings, and higher-order polytopes remains future work.
- Domain Focus: While trained on broad data, evaluation was conducted almost exclusively on speech; extensive validation on music and acoustic event domains is pending.
- Downstream Integration: Generative downstream tasks (e.g., integrating Q2D2 tokens into Text-to-Speech LLMs) were not evaluated in this study.
Summary updated