Test-Time Steering for Lossless Text Compression via Weighted Product of Experts

Qihang Zhang, Muchen Li, Ziao Wang, Renjie Liao, Lele Wang

TL;DR

This paper introduces a novel "Test-Time Steering" (TTS) framework called Weighted Product of Experts (wPoE) for lossless text compression. wPoE adaptively combines a powerful but generalization-prone neural language model with a simple, training-free universal compressor at inference time, achieving better compression rates than either model alone without requiring any retraining.

What Problem Does This Paper Address?

Lossless text compression is vital for handling the ever-growing volume of digital data. Traditional compressors like gzip are fast and reliable across different data types but often fall short of the compression ratios achieved by modern neural network-based compressors. These neural compressors, especially those leveraging Large Language Models (LLMs), can model data distributions more effectively and achieve superior compression. However, they suffer from a significant drawback: poor generalization to data distributions they weren't trained on, and they often require computationally expensive fine-tuning. This paper aims to bridge this gap by developing a method that combines the strengths of both approaches – the high compression potential of neural models and the robust generalization of traditional methods – without the need for costly retraining.

Methods

The core innovation is the Weighted Product of Experts (wPoE) framework. At inference time, wPoE dynamically combines the predictions of multiple "experts" to generate a steered probability distribution for the text. The key idea is to use a pretrained neural language model (the "strong" expert) and a simple, training-free universal compressor (like Naive Bayes with Laplace smoothing, the "weak" but generalizable expert) as the components.

The wPoE model calculates the probability of the next token by taking a weighted product of the probabilities assigned by each expert, then normalizing the result. The crucial part is that the weights are learned at test time using a small amount of data from the target distribution. This "test-time steering" allows the model to adapt to new data distributions without any gradient updates to the pretrained neural model itself. The paper also provides a theoretical guarantee that the wPoE model will perform at least as well as the best individual expert.

Datasets

The paper evaluates its approach on several diverse datasets:

  • enwik8 and enwik9: Standard benchmarks for byte-level compression, derived from Wikipedia.
  • code: A newly introduced benchmark for code generation, comprising Python programming tasks across various libraries and domains. This dataset is designed to be significantly different from natural language corpora.
  • math: A dataset of challenging mathematics problems from Hendrycks et al. (2021).
  • shakespeare: The complete works of William Shakespeare.

For training, the paper mentions using the enwik8 dataset for the byte-level decoder-only Transformers. For evaluation, all datasets are partitioned into sequences of 2048 bytes.

Key Insights and Ideas

  • Test-Time Steering is Powerful: The ability to adapt a model's behavior at inference time using a small amount of target data, without retraining, is a significant advantage for generalization.
  • Complementary Experts: Combining a powerful, specialized neural model with a simple, general-purpose model can yield performance superior to either alone, especially when dealing with out-of-distribution data.
  • Naive Bayes as a Generalization Anchor: The simple Naive Bayes classifier with Laplace smoothing acts as a surprisingly effective "anchor" for generalization, preventing the combined model from overfitting to the target distribution.
  • Minimal Data for Adaptation: The paper demonstrates that a near-optimal adaptation weight can be found using just a single data point from the target distribution, making the adaptation process extremely efficient.
  • Theoretical Guarantees Matter: The theoretical proof that wPoE performs at least as well as the best expert provides confidence in its robustness.

Key Results

  • Consistent Improvements: The wPoE framework consistently improved compression rates across all tested datasets and models, especially on out-of-distribution (OOD) data.
  • Significant Gains on OOD Data: For the "code" dataset, which is highly OOD for models trained on natural language, the wPoE approach showed substantial improvements. For example, Transformer 200K + wPoE achieved a compression rate of 53.94% compared to 65.67% for the base Transformer.
  • Efficiency: The adaptation process (learning the optimal weight) is computationally very cheap, requiring only a few iterations with a single data point, significantly less than fine-tuning.
  • Scalability: The approach shows benefits across various model sizes, from smaller byte-level Transformers to larger LLMs like GPT-2 and LLaMA 3.
  • Multiple Experts: Incorporating more experts (e.g., multiple Transformers plus Naive Bayes) further boosted performance, with the Naive Bayes expert providing a particularly significant gain.

Limitations and Future Work

  • Limited Expressive Capacity of Weights: The model's adaptability is primarily controlled by a single scalar weight (or a few weights if more experts are used). This limits its expressive capacity compared to full fine-tuning.
  • Expert Diversity is Crucial: The wPoE approach relies on the experts being sufficiently diverse. If experts are too similar, the stronger one might dominate, and the weaker one's contribution will be negligible.
  • Future Work: The authors suggest exploring other suitable experts that could further enhance generalization across various NLP tasks. Identifying effective priors for specific tasks and integrating them with pretrained models is a promising direction for future research. They also briefly explored dynamic, per-token weights, which showed modest gains but came with increased complexity.

Summary updated

Abstract

Lossless compression techniques are crucial in an era of rapidly growing data. Traditional universal compressors like gzip offer low computational overhead, high speed, and broad applicability across data distributions. However, they often lead to worse compression rates than modern neural compressors, which leverage large-scale training data to model data distributions more effectively. Despite their advantages, neural compressors struggle to generalize to unseen data. To address this limitation, we propose a novel framework that performs Test-Time Steering via a Weighted Product of Experts (wPoE). At inference, our method adaptively combines a universal compression model with a pretrained neural language model, ensuring the compression rate is at least as good as that of the best individual model. Extensive experiments demonstrate that our approach improves the performance of text compression without requiring fine-tuning. Furthermore, it seamlessly integrates with any autoregressive language model, providing a practical solution for enhancing text compression across diverse data distributions.
BibTeX
@inproceedings{zhang2025test,
  author = {Qihang Zhang and Muchen Li and Ziao Wang and Renjie Liao and Lele Wang},
  title = {{Test-Time Steering for Lossless Text Compression via Weighted Product of Experts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure