Language models that read audio as raw bytes turn out to be very good lossless compressors. They split each digital audio sample into its bytes, most significant first, and predict one byte at a time from a fixed vocabulary of 256 values. At 8 and 16 bits per sample they beat classical codecs, but at studio 24-bit resolution they lose, because the bottom byte is mostly recording noise that no model can predict. This note proposes that a byte-level audio model should model autoregressively only the bits a listener can actually hear in each short frame. The remaining bits would go to a cheap noise model. The dividing line, the "audible depth," would be set frame by frame from a standard auditory masking estimate, not from how hard the bits are to predict.
The same byte hierarchy also gives a controlled testbed for a problem that runs through generative-audio evaluation: objective scores and human judgments disagree. Removing low-order bits adds degradation that is exactly known, spectrally white, and monotone in dose. That lets us ask which metrics follow the point where humans first notice a change, and which keep reporting improvements that nobody can hear. The two halves of the proposal depend on each other. The method assumes a perceptual floor exists, and the measurement study checks whether it does.
Core Claim
Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio (Trilobyte) reports that a byte-level Transformer with an arithmetic coder beats FLAC by 217% at 8-bit and 18% at 16-bit, but trails it by 9% at 24-bit. The authors attribute the loss to low-order bytes that are "imperceptible noise." That attribution was never tested perceptually. Our claim has two parts.
First, next-byte cross-entropy is itself a perceptually misaligned objective. Most of its loss and gradient are spent on bit planes that listeners cannot detect, and the audibility floor moves over time. In a loud passage, masking hides many low bits. In a quiet fade, the same bits become audible.
Second, a model that splits each frame at a perceptually computed audible depth should do better than both a fixed truncation and a split chosen by predictability alone. It should reach the same perceptual quality with fewer modeled bytes and fewer compute operations, and it should generate continuations that listeners rate higher.
For a frame $t$, let $M_t(f)$ be the masking threshold, meaning the power below which added noise at frequency $f$ is inaudible, computed from the frame's own spectrum with a standard psychoacoustic model. Let $F$ be the number of frequency bins. Quantizing to $d$ bits with step $\Delta_d = 2^{1-d}$ (full scale normalized to $\pm 1$) adds white noise of power $\Delta_d^2/12$ spread evenly over the bins. The audible depth is the smallest bit depth whose noise stays under the mask in every bin:
$$d_t = \min \left\{ d : \frac{\Delta_d^2}{12F} \le M_t(f) \ \text{for all } f \right\}.$$
Bits above $d_t$ are modeled by the byte Transformer. Bits below $d_t$ are handled by a parametric noise model, such as a Laplace distribution whose scale depends on context. That model codes them for lossless use and samples them for generation.
Relation to Prior Work
Two byte-modeling papers supply the machinery. Byte Latent Transformer: Patches Scale Better Than Tokens groups text bytes into variable-length patches, placing a boundary wherever a small model finds the next byte hard to predict. It matches BPE models up to 8B parameters with fewer inference FLOPs. MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers pairs a large model over fixed patches with a small model inside each patch, which makes million-byte contexts affordable. We reuse the MEGABYTE global–local design so that audio sequences fit in context.
We deliberately break from the BLT rule. In audio, the least predictable bytes are exactly the inaudible ones, so a predictability-based rule sends compute to the wrong place. This is the central contrast we test.
Trilobyte motivates the work: it found the 24-bit failure. It also constrains the design: a general text-trained byte model does not transfer to audio, so everything is trained on audio from scratch.
On the evaluation side, several papers motivate the measurement study by documenting divergence between metrics and listeners. Geometric Iterative Retrieval for Neural Audio Codec Resynthesis found that log-spectral distance stops improving after 3 of 9 codec quantization layers, while listeners prefer the full 9-layer decode. Those codec layers change spectrum, phase and texture together, so that finding cannot say which property the metric misses. Byte-plane truncation changes only one variable, white noise level, which fixes that confound.
Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms reports correlation with human ratings of 0.52 for its distributional embedding distance and 0.39 for signal-to-distortion ratio on MagnaTagATune. ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric is a reference-based perceptual metric; we use it as the automated perceptual score and test whether it tracks human detection thresholds on this ladder. SongEval: A Benchmark Dataset for Song Aesthetics Evaluation and CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction concern song-level aesthetics. They sit at a much coarser scale than ours. We treat them as out of scope, except as a sanity check that the method does not damage musical content.
Research Program
Measurement. We first build the byte-plane ladder. Clips from the Trilobyte suite are requantized at every depth from 24 down to 8 bits, with triangular dither applied so that the error is white and independent of the signal. For each clip we compute the per-frame audible depth $d_t$. We then collect human detection thresholds with ABX tests (a listener hears references A and B and says which one a third clip X matches) at each depth. In parallel we record how each objective score responds: the byte model's bits per byte on each plane, log-spectral distance, ViSQOL, and embedding distance.
This yields three answers: whether the per-frame masking estimate predicts where listeners' detection falls, which metrics bend at the human threshold, and whether Trilobyte's "imperceptible LSB" claim holds for 24-bit masters.
Training dynamics. We train a Trilobyte-scale model (90–140M parameters) with a MEGABYTE-style patch hierarchy and log the loss and gradient norm per byte position. This quantifies how much training signal goes to planes below $d_t$.
Method. We then train the audible-depth model. Each frame carries its $d_t$ as a prefix token. For lossless or near-lossless coding, the encoder computes $d_t$ and sends it as side information. For generation, the model predicts $d_t$ itself. Planes above $d_t$ are modeled autoregressively. Planes below it are coded or sampled by the noise model, conditioned on the high-plane context. The same model supports three modes:
- Lossless coding: every plane is coded, and the noise model handles those below $d_t$.
- Perceptually lossless coding: planes below $d_t$ are dropped at the encoder, and the decoder replaces them with shaped dither.
- Continuation generation from a prompt.
Baselines
Naive reference. Trilobyte as published: full-depth byte modeling with plain cross-entropy on all planes. We retrain it on our splits with the same architecture, data and token budget, since we cannot rely on its reported averages alone. Its reported results, 18% better than FLAC at 16-bit and 9% worse at 24-bit, are the numbers to reproduce before any comparison is trusted. Beating it means achieving lower bits per sample in lossless mode at 24-bit, or fewer modeled bytes at equal ABX transparency.
Fixed truncation. Requantize everything to 16 bits with dither, then run Trilobyte. This is the strongest simple competitor, because 16-bit is where byte models already win. The comparison uses the same model size and training steps. The audible-depth model must match its ABX transparency while modeling fewer bytes on average, or it must stay transparent on quiet material where fixed 16-bit fails.
Fixed-split hybrid. A byte model on the top two bytes and a context Laplace coder on the bottom byte. This competes directly for the lossless 24-bit number, and may well close most of Trilobyte's gap by itself. It gets the same noise-model capacity as our method. In lossless mode, ours wins only if a per-frame split beats the fixed one in bits per sample.
Codec tokens. A byte-free comparison using Descript Audio Codec tokens modeled at matched bitrate, evaluated on ViSQOL and ABX. Trilobyte reports that neural codecs do worse than FLAC as full-fidelity representations.
Ablations. Two ablations isolate the key ingredient. The first uses a BLT-style predictability split: the split depth is set where the model's next-byte entropy crosses a threshold, tuned to the same average depth. The second uses a shuffled audible-depth map, with the same distribution of $d_t$ values assigned to the wrong frames. If perceptual placement matters, both should lose to the true $d_t$ at equal average depth.
There is no reported result on Lacuna for byte-level models at a perceptual-transparency operating point, so those comparisons are established within this study.
Why not just…?
Why not just truncate to 16 bits and move on? This is the most serious threat. If listeners cannot tell 16-bit from 24-bit on these corpora, a global truncation captures most of the benefit. The expected failure is on quiet passages, fades and solo piano. There the masking threshold drops, and even 16-bit dithered noise may become audible, while loud dense mixes could go well below 16 bits. The ladder study settles this directly by measuring how per-frame audible depth varies. If $d_t$ barely varies within and across clips, we will report that fixed truncation suffices and drop the adaptive method.
Why not just use neural codec tokens, which are 50–100× shorter? Codec tokens are the right choice for low-bitrate generation. They do not reach full fidelity: Trilobyte found them worse than FLAC as full-fidelity bases, and the Geometric Iterative Retrieval work shows that their fine layers carry perceptually important smoothness that metrics fail to register. Our target is the near-transparent regime, which codecs do not reach. The codec baseline tests that boundary.
Why not just apply BLT entropy patching to audio bytes? A predictability rule puts boundaries and compute exactly where noise lives, the opposite of what perception needs. That is our hypothesis, not established fact. The predictability-split ablation at matched average depth is the direct test, and a tie would mean the perceptual criterion adds nothing.
Why not just use a conventional perceptual transform codec? For pure compression at moderate bitrates, masking-based transform codecs will likely win, and we do not claim otherwise. The contribution is a single byte-level model that is lossless when needed, perceptually lossless when allowed, and generative. A transform codec's bitstream is not something a byte model can easily model or continue. If a reader cares only about compression bitrate, this proposal is not the answer.
Proposed Experiments
The data are the Trilobyte suite at 16 and 24 bits: MusDB18, Beethoven sonatas, the commercial high-resolution music set, VCTK, LibriSpeech and Birdvox.
Check 1: does the masking estimate predict hearing? Per-frame $d_t$ should predict ABX detection depth better than a clip-constant depth. We will use roughly 20 trained listeners, with forced-choice thresholds per clip. A negative result here falsifies the premise and ends the method track.
Check 2: which metrics track the threshold? We measure the dose–response of bits per byte, log-spectral distance, ViSQOL and embedding distance along the ladder. The question is whether each metric keeps changing below the human detection threshold, the byte-level analogue of the 3-versus-9-layer mismatch. We report where each curve bends relative to the threshold.
Check 3: does the method beat its competitors? At matched parameters and tokens, the audible-depth model is compared against fixed truncation, the fixed-split hybrid, the predictability split and the shuffled map. Measures are lossless bits per sample, bytes and FLOPs per second of audio at ABX transparency, and MUSHRA ratings (a standard listening test where listeners score several versions against a reference) for 10-second piano and speech continuations. Embedding distance and ViSQOL are reported alongside so the paper can show where they agree or disagree with listeners.
The result that would justify the work is the true $d_t$ beating the shuffled map and the predictability split at equal average depth, in both modeled bytes at transparency and listener ratings of generated audio.
References
- Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio
- Byte Latent Transformer: Patches Scale Better Than Tokens
- MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
- Geometric Iterative Retrieval for Neural Audio Codec Resynthesis
- Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms
- ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction