Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio

Phillip Long, Zachary Novack, Chris Donahue

TL;DR

While autoregressive language models (LMs) excel at compressing low-fidelity 8-bit audio, scaling them to full-fidelity (16/24-bit) audio triggers an exponential vocabulary explosion. This paper introduces Trilobyte, a hierarchical byte-level tokenization scheme that keeps vocabulary size constant, enabling the first tractable 24-bit LM-based lossless audio compressor. However, comprehensive benchmarking reveals a sobering truth: as audio fidelity increases, the performance gap between neural models and traditional codecs like FLAC narrows significantly.

What Problem Does This Paper Address?

Repurposing autoregressive language models for lossless data compression is a powerful paradigm. By pairing an LM's next-token probability distributions with an arithmetic coder, a file can be compressed to a size that approaches the model's cross-entropy loss.

While this works remarkably well for text and 8-bit audio, real-world audio applications demand "CD-quality" (16-bit, 44.1kHz) or professional production quality (24-bit). Directly applying standard sample-level tokenization to these bit depths causes an exponential vocabulary explosion:

  • 8-bit: 256 tokens (manageable)
  • 16-bit: 65,536 tokens (prohibitive)
  • 24-bit: 16,777,216 tokens (completely intractable)

At 24-bit, the embedding and output projection layers alone would require billions of parameters, rendering standard autoregressive approaches computationally impossible. This paper addresses this bottleneck, exploring whether LM-based compression can scale to full-fidelity audio and truly compete with traditional codecs.

Methods

To bypass the exponential vocabulary scaling $O(2^b)$, the authors propose Trilobyte, a hierarchical byte-level tokenization scheme.

Instead of treating an entire $b$-bit audio sample as a single token, Trilobyte decomposes each sample into its constituent bytes ($B = \lceil b/8 \rceil$). It then predicts over a fixed vocabulary of 256 possible values at each byte position. By interleaving these bytes from the most significant byte (MSB) to the least significant byte (LSB), the model implicitly learns separate byte distributions through its autoregressive context, keeping the vocabulary size at a constant $O(1)$.

Figure 1: Tokenization strategies for language model compression. Standard sample-level tokenization (top) yields vocabulary size |V| = 2b. This exponential scaling inhibits modeling of industry-standard bit depths (16, 24). Trilobyte’s hierarchical byte-level tokenization (bottom) decomposes samples into bytes, yielding constant |V| = 256 regardless of bit depth (at the cost of increasing sequence length by ⌈b/8⌉). Both feed into an AR LM and arithmetic coder, but Trilobyte enables tractable 24-bit modeling.

The interleaved byte sequences are fed into a standard decoder-only Transformer (GPT-2 architecture) paired with an arithmetic coder.

Additionally, the authors introduce a multi-bit-rate masking scheme for transfer learning. By randomly masking out lower-significance bytes with a learned null token during training, a single generalist Trilobyte model can be trained to compress audio at arbitrary bit depths (8-, 16-, or 24-bit) without retraining.

Datasets

The authors assembled a diverse evaluation suite spanning multiple audio domains and sampling rates (16kHz–48kHz):

  • Music: MusDB18 (multi-track mixes and stems), Beethoven (piano sonatas), YouTube Mix (solo piano), and a newly introduced Commercial Music Dataset containing 1,569 songs at 16-bit and 933 songs at 24-bit (including high-resolution recordings up to 192kHz).
  • Speech: LibriSpeech (clean English speech), LJSpeech (single-speaker audiobook), SC09 (spoken digits), and VCTK (multi-speaker English).
  • Bioacoustics: Birdvox (bioacoustic bird vocalizations).
  • Sound Effects: Epidemic Sound (sound effects library).

Key Insights and Ideas

  • Bit Depth is the Ultimate Bottleneck: Sampling rate and data domain have minor impacts on compression scaling, but increasing bit depth drastically diminishes the performance advantages of LMs over traditional codecs.
  • The 24-bit Noise Wall: At 24 bits, the least significant bits largely consist of imperceptible, low-amplitude acoustic noise. Because this noise behaves like random data, LMs struggle to find patterns to compress. Traditional codecs like FLAC use simple Rice coding, which happens to be nearly optimal for handling this geometric noise distribution.
  • Why Neural Audio Codecs (NACs) Fail at Lossless Compression: In baseline experiments, the authors tried replacing FLAC's linear predictive coding with discrete latent codes from Neural Audio Codecs (like DAC or EnCodec) and compressing the residuals. This approach performed poorly because NACs are trained on perceptual losses (making things sound good to humans) rather than minimizing absolute waveform error. This causes their residuals to be uniformly distributed rather than geometrically distributed, making subsequent Rice coding highly inefficient.

Key Results

The benchmarked compression rates ($R = \text{Original Size} / \text{Compressed Size}$) revealed distinct performance tiers based on bit depth:

  • 8-bit Regime: LMs substantially outperform FLAC, yielding a 217% average improvement. For instance, on the Beethoven dataset, Trilobyte achieved a 7.94x compression rate compared to FLAC's 1.69x.
  • 16-bit Regime: The gap narrows. Trilobyte still consistently beats FLAC, but with a more modest 18% average improvement (e.g., 2.82x vs 2.15x on MusDB18 Mono).
  • 24-bit Regime: FLAC reclaims the crown. Trilobyte achieves a 1.48x compression rate, falling 9% short of FLAC's 1.63x.
  • Generalist Transfer Model: The single generalist Trilobyte model trained with byte-masking performed remarkably well, achieving 1.49x (24-bit), 1.78x (16-bit), and 3.4x (8-bit) compression rates on commercial music, proving that a single model can handle multi-fidelity audio compression.

Limitations and Future Work

The most prominent limitation is computational speed. Autoregressive language models require sequential generation and are orders of magnitude slower than traditional codecs. The modest compression gains at 16-bit are currently difficult to justify given the immense computational overhead of running inference on Transformers.

Furthermore, the model size itself introduces storage overhead; a billion-parameter model requires gigabytes of space, meaning the system only provides a net storage benefit when compressing massive, large-scale audio archives. Future work will need to focus on improving the efficiency of these architectures and finding better ways to model or filter out the random noise inherent in 24-bit audio.

Summary updated

Abstract

Autoregressive "language" models (LMs) trained on raw waveforms can be repurposed for lossless audio compression, but prior work is limited to 8-bit audio, leaving open whether such approaches work for practical settings (16/24-bit) and can compete with existing codecs. We benchmark LM-based compression on full-fidelity audio across diverse domains (music, speech, bioacoustics), sampling rates (16kHz-48kHz), and bit depths (8, 16, 24-bit). Standard sample-level tokenization becomes intractable at higher bit depths due to vocabulary size (65K for 16-bit; 16.7M for 24-bit). We propose Trilobyte, a byte-level tokenization schema for full resolution audio, improving vocabulary scaling from $O(2^{b})$ to $O(1)$ and enabling the first tractable 24-bit LM-based lossless compression. While LMs consistently outperform FLAC and yield state-of-the-art compression at 8-bit and 16-bit, we observe that compression gains become more modest as bit depth increases beyond 8-bit.
BibTeX
@misc{long2026benchmarking,
  author = {Phillip Long and Zachary Novack and Chris Donahue},
  title = {{Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio}},
  year = {2026},
  eprint = {2603.08683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.08683},
}
Figure