Large Language Models and Normalized Compression Distance: Better Compression Yet Worse Accuracy

John Hurwitz, Charles Nicholas, Edward RaffFrontiers in Artificial Intelligence and Applications 2025 · DOI · OpenAlex

Abstract

It is generally well understood that predictive classification and compression are intrinsically related concepts in information theory. Indeed, many deep learning methods are explained as learning a kind of compression, and that better compression leads to better performance. We interrogate this hypothesis via the Normalized Compression Distance (NCD), which explicitly relies on compression as the means of measuring similarity between sequences and thus enables nearest-neighbor classification. By turning popular large language models (LLMs) into lossless compressors, we develop a Neural NCD and compare LLMs to classic general-purpose algorithms like gzip in a compression-distance-based classification setting. We find that whether neural compressors achieve better accuracy over a gzip baseline is dataset-dependent, despite consistently superior compression ratios. Though the best neural compressor achieves up to 7.49 times better compression, we observe an up to 22.2% decrease in accuracy relative to the gzip baseline.
BibTeX
@inproceedings{hurwitz2025large,
  author = {John Hurwitz and Charles Nicholas and Edward Raff},
  title = {{Large Language Models and Normalized Compression Distance: Better Compression Yet Worse Accuracy}},
  booktitle = {Frontiers in Artificial Intelligence and Applications},
  year = {2025},
  doi = {10.3233/faia251322},
}

Loading related content...

Figure