Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio

P. E. Long, Zachary Novack, Chris Donahue

TL;DR

Researchers have introduced Trilobyte, a byte-level tokenization scheme that allows standard language models (LMs) to perform lossless audio compression at professional bit depths (16-bit and 24-bit). While these models significantly outperform the industry-standard FLAC at lower resolutions, the performance gap narrows at higher fidelities, revealing that bit depth—rather than sampling rate or data domain—is the primary bottleneck for neural audio compression.

What Problem Does This Paper Address?

Lossless audio compression is dominated by traditional codecs like FLAC, which use linear predictive coding to guess the next sample and then store the "error" (residual). While researchers have recently used Autoregressive (AR) language models to achieve massive gains in lossy compression, applying them to lossless audio has been stuck in the "8-bit era."

The hurdle is the vocabulary explosion. In a standard LM, every possible audio sample value is treated as a unique "token." For 8-bit audio, there are only 256 possible values—easy for a model to handle. But for CD-quality (16-bit), there are 65,536 values, and for professional 24-bit audio, there are over 16.7 million. Building a model with a 16-million-token vocabulary is computationally intractable. Consequently, we didn't know if LMs could actually scale to the high-fidelity audio used by professionals.

Methods

The authors propose Trilobyte, a hierarchical byte-level tokenization scheme. Instead of treating a 16-bit or 24-bit sample as one giant token, Trilobyte decomposes it into individual bytes (e.g., a 24-bit sample becomes 3 bytes).

Key technical details include:

  • Constant Vocabulary: By modeling at the byte level, the vocabulary size stays fixed at 256, regardless of the audio's bit depth.
  • Interleaving: The model predicts bytes in order from Most Significant Bit (MSB) to Least Significant Bit (LSB).
  • Architecture: They used standard decoder-only Transformers (GPT-2 style) and paired them with an arithmetic coder to turn the model's probability distributions into a compressed bitstream.
  • Transfer Learning: They introduced a "masking" scheme during training, allowing a single model to compress audio at any bit depth (8, 16, or 24) by learning a special "null" token for lower-significance bytes.

Datasets

The study benchmarks across a massive variety of audio domains to ensure the results aren't specific to just one type of sound:

  • Music: MusDB18 (multi-track), Beethoven piano sonatas, YouTube Mix, and a new Commercial Music dataset (1,569 songs) to reflect real-world high-resolution audio.
  • Speech: LibriSpeech, LJSpeech, SC09 (digits), and VCTK (multi-speaker).
  • Other: Birdvox (bioacoustic bird calls) and Epidemic Sound (sound effects).

Key Insights and Ideas

  • The Bit Depth Bottleneck: The most significant finding is that LMs struggle more as bit depth increases. At 8-bit, LMs are "magic," beating FLAC by over 200%. At 16-bit, the gain drops to 18%. At 24-bit, the LM actually loses to FLAC by 9%.
  • The "Noise" Floor: The authors suggest that at 24-bit, the least significant bits are essentially imperceptible noise. Traditional math-based coding (Rice coding in FLAC) is nearly optimal for this kind of noise, whereas LMs try to find patterns where none exist.
  • Text Models Don't Speak Audio: They tested Llama-2 (a model trained on text) for audio compression. It performed poorly, suggesting that general-purpose text LMs don't naturally "understand" the statistical structure of raw audio waveforms.
  • Neural Codecs vs. Lossless: They also tried using popular neural audio codecs (like EnCodec or DAC) as a base for lossless compression. Surprisingly, these performed worse than FLAC because their reconstruction errors don't follow the "geometric distribution" that standard entropy coders expect.

Key Results

  • 8-bit Performance: Trilobyte achieved a 217% average improvement over FLAC. On specific datasets like Beethoven piano recordings, it reached nearly 8x compression.
  • 16-bit Performance: The model remained state-of-the-art with an 18% average improvement over FLAC across all domains.
  • 24-bit Performance: This was the first time 24-bit LM compression was even possible. While it trailed FLAC by 9%, it proved that byte-level modeling is a viable path forward for high-resolution tasks.
  • Generalization: A single "Transfer" model trained on all datasets performed nearly as well as models trained on specific domains, proving that a "universal" lossless audio compressor is feasible.

Limitations and Future Work

The primary limitation is speed. While FLAC is incredibly fast and lightweight, running a 90M-140M parameter Transformer to decompress audio is computationally expensive and slow.

Future work may involve:

  • Optimizing the architecture for faster inference (e.g., using Linear Transformers or State Space Models).
  • Investigating why LMs struggle with the "noise" in 24-bit audio and whether hybrid approaches (LM for MSBs, traditional coding for LSBs) could yield better results.
  • Scaling the models further to see if larger parameter counts can close the gap at 24-bit.

Summary updated

Abstract

Autoregressive "language" models (LMs) trained on raw waveforms can be repurposed for lossless audio compression, but prior work is limited to 8-bit audio, leaving open whether such approaches work for practical settings (16/24-bit) and can compete with existing codecs. We benchmark LM-based compression on full-fidelity audio across diverse domains (music, speech, bioacoustics), sampling rates (16kHz-48kHz), and bit depths (8, 16, 24-bit). Standard sample-level tokenization becomes intractable at higher bit depths due to vocabulary size (65K for 16-bit; 16.7M for 24-bit). We propose Trilobyte, a byte-level tokenization schema for full resolution audio, improving vocabulary scaling from $O(2^{b})$ to $O(1)$ and enabling the first tractable 24-bit LM-based lossless compression. While LMs consistently outperform FLAC and yield state-of-the-art compression at 8-bit and 16-bit, we observe that compression gains become more modest as bit depth increases beyond 8-bit.
BibTeX
@misc{long2026benchmarking,
  author = {P. E. Long and Zachary Novack and Chris Donahue},
  title = {{Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio}},
  year = {2026},
  eprint = {2603.08683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.08683},
}
Figure