Two-dimensional quantization for geometry-aware audio coding

Tal Shuster, Eliya Nachmani

TL;DR

This paper introduces Q2D2, a novel two-dimensional quantization method for neural audio codecs. By projecting feature pairs onto structured 2D grids (like hexagons or rhombuses), Q2D2 captures feature correlations more effectively than existing methods, leading to improved compression efficiency, higher codebook utilization, and state-of-the-art audio reconstruction quality with fewer tokens.

What Problem Does This Paper Address?

Modern neural audio codecs, crucial for tasks like speech synthesis and music generation, rely on quantization to compress audio signals into discrete tokens. Existing methods like Vector Quantization (VQ) and Residual Vector Quantization (RVQ) often struggle with inefficient codebook utilization and fail to capture the geometric relationships within the latent space. Finite Scalar Quantization (FSQ) offers simplicity and high utilization but treats each feature dimension independently, missing out on valuable inter-channel correlations. This leads to a trade-off between compression efficiency and reconstruction quality. The authors aim to bridge this gap by introducing a quantization scheme that leverages geometric structures to capture these correlations without sacrificing the simplicity and efficiency of methods like FSQ.

Methods

The core innovation is Two-Dimensional Quantization (Q2D2). Here's how it works:

  • Feature Projection: The output of the neural encoder is projected into a lower-dimensional space and then bounded to an interval (e.g., [-1, 1]).
  • Pairing and Gridding: These dimensions are then grouped into pairs. Each pair is projected onto a structured 2D grid (hexagonal, rectangular, or rhombic).
  • Quantization: The feature pair is quantized to the nearest point on this 2D grid. This creates an implicit codebook defined by the grid structure and the number of levels per dimension, avoiding the need to learn a large explicit codebook.
  • Straight-Through Estimator (STE): STE is used to enable gradient propagation during training, ensuring differentiability and stability.
  • Lightweight Projections: Simple linear projections are used to map features into and out of the quantization space, keeping the model lightweight.

This approach contrasts with:

  • VQ: Learns an explicit codebook and quantizes vectors in a high-dimensional space, often leading to underutilized codebooks.
  • FSQ: Quantizes each dimension independently on a 1D grid, missing inter-channel correlations.
  • RVQ: Uses multiple layers of quantization, increasing complexity and potential for inefficiency.

Datasets

The paper uses several datasets for training and evaluation:

  • Training:
    • LibriTTS (approx. 585 hours) for ablation studies.
  • LibriTTS, VCTK, CommonVoice (approx. 8K hours total) for main experiments.
  • Emilia (multilingual) and MLS (multilingual) (approx. 150K hours) for evaluating against other baselines.
  • AudioSet (approx. 2000 hours) for general audio.
  • Jamendo and MUSDB18 for music.
  • Evaluation:
    • LibriTTS test-clean, test-other.
  • LJSpeech.
  • LibriSpeech test-clean.
  • ARCH benchmark (12 datasets across speech, music, audio domains) for semantic representation evaluation.

No new benchmarks were introduced; the authors leverage existing datasets and evaluation protocols.

Key Insights and Ideas

  • Geometry Matters in Latent Space: The paper strongly suggests that the geometric arrangement of quantization levels in the latent space is not just an aesthetic choice but directly impacts how well correlations between features are captured.
  • Rhombic Grids Offer Superior Packing: Rhombic grids consistently outperform hexagonal and rectangular grids in reconstruction metrics. This is attributed to their "oblique basis" and "space-filling partition," which better align with learned latent distributions and lead to more uniform coverage and lower quantization error.
  • Implicit Codebooks are Efficient: By defining codebooks implicitly through geometric grids, Q2D2 avoids the significant memory and computational overhead of learning large, explicit codebooks, as seen in traditional VQ.
  • Moderate Dimensionality is Key: Ablation studies show that a moderate dimensionality (e.g., 6 dimensions) for the projected features strikes the optimal balance between compactness and representational capacity, with higher or lower dimensions degrading performance.
  • STE is Sufficient: Despite the complexity of 2D quantization, the standard Straight-Through Estimator (STE) proves to be the most effective and stable method for gradient propagation, simplifying the training process.

Key Results

  • State-of-the-Art Reconstruction: Q2D2 achieves competitive to superior reconstruction quality across speech, audio, and music domains, often surpassing existing state-of-the-art (SOTA) models.
  • Low Token Rates: Q2D2 achieves comparable or better quality with significantly fewer tokens per second (e.g., 166 tokens at 3.3 kbps vs. 900 tokens for DAC at 9.0 kbps).
  • High Codebook Utilization: The implicit grid-based approach ensures high codebook utilization without needing auxiliary tricks like commitment losses.
  • Efficiency: Q2D2 demonstrates minimal overhead in terms of runtime and memory, with RTF comparable to WavTokenizer.
  • Semantic Representation: Q2D2 shows strong generalization capabilities on the ARCH benchmark, outperforming DAC and Encodec in semantic representation tasks.
  • Subjective Quality: MUSHRA evaluations show Q2D2 at 3.3 kbps outperforming SOTA DAC at 9.0 kbps, indicating excellent perceptual quality.

Limitations and Future Work

  • Limitations:
    • The paper focuses primarily on 2D quantization. While promising, it acknowledges that higher-dimensional structures might capture even richer correlations.
  • While robust, the paper notes that certain choices (like grid type and level selection) still involve trade-offs between bitrate, utilization, and quality.
  • Future Work:
    • Extend to 3D: The authors propose extending Q2D2 to three-dimensional quantization geometries and exploring higher-order polytopes and mixed-dimensional partitions.
  • Explore Simplex Tilings: Investigating simplex tilings and other complex grid structures could further enhance representational capacity.
  • Audio and Music Domains: While experiments cover speech, audio, and music, further dedicated study in the latter two domains is suggested.

Summary updated

Abstract

Recent neural audio codecs have achieved impressive reconstruction quality, typically relying on quantization methods such as Residual Vector Quantization (RVQ), Vector Quantization (VQ) and Finite Scalar Quantization (FSQ). However, these quantization techniques limit the geometric structure of the latent space, make it harder to capture correlations between features leading to inefficiency in representation learning, codebook utilization and token rate. In this paper we introduce Two-Dimensional Quantization (Q2D2), a quantization scheme in which feature pairs are projected onto structured 2D grids, such as hexagonal, rhombic, or rectangular tiling and quantized to the nearest grid values, yielding an implicit codebook defined by the product of grid levels, with codebook sizes comparable to conventional methods. Despite its simple geometric formulation, Q2D2 improves audio compression efficiency, with low token rates and high codebook utilization while maintaining state of the art reconstruction quality. Specifically, Q2D2 achieves competitive to superior performance in various objective and subjective reconstruction metrics, across extensive experiments in speech, audio and music domains compared to state of the art models. Comprehensive ablation studies further confirm the effectiveness of our design choices.
BibTeX
@inproceedings{shuster2026two,
  author = {Tal Shuster and Eliya Nachmani},
  title = {{Two-dimensional quantization for geometry-aware audio coding}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=Ja9jo9pDvq},
}
Figure