When deploying large statistical models—whether they are n-gram language models, deep convolutional networks, or modern large language models (LLMs)—we inevitably face memory and latency constraints. The naive approach to compression is to drop parameters based on simple heuristics like magnitude or frequency. For instance, one might prune rare words from a vocabulary or zero out the smallest weights in a neural network.
However, frequency and magnitude are poor proxies for informational value. A rare parameter or a small weight can still be mathematically vital to the shape of a model's output. If we blindly truncate based on counts, we risk destroying accuracy on rare but critical edge cases.
Information-theoretic pruning solves this by framing model compression as a distributional optimization problem: how do we remove components such that the probability distribution of the compressed model diverges as little as possible from the original model? By treating parameters not as isolated numbers, but as contributors to a global probability distribution, we can achieve massive compression ratios while preserving predictive power.
Minimizing Relative Entropy
The most direct way to measure the impact of removing a parameter is to calculate the Kullback-Leibler (KL) divergence—or relative entropy—between the original model's output distribution $P$ and the pruned distribution $Q$. If $D_{\text{KL}}(P \parallel Q)$ is close to zero, the behavioral difference between the two models is negligible, meaning the removed parameter was redundant.
This principle was pioneered in early natural language processing to compress massive language models. Kae-Cherng Yang et al. demonstrated in Truncation on Combined Word-Based and Class-Based Language Model Using Kullback-Leibler Distance Criterion that pruning parameters based on KL distance preserves model accuracy far more effectively than frequency thresholds. This approach successfully identifies and retains rare sequences that are mathematically vital to the distribution's overall shape.
Original Distribution (P) ---> [ Pruning Candidate ]
|
Does removal shift
the distribution shape?
|
Calculate D_KL(P || Q)
|
Sparsified Distribution (Q) <--- [ Low Divergence = Safe to Prune ]
In n-gram language modeling, this strategy is often operationalized by evaluating whether a higher-order sequence adds enough information over its lower-order backoff model. Kenneth Church et al. leveraged relative entropy pruning in Compressing Trigram Language Models With Golomb Coding to discard n-grams that failed to significantly alter the probability distribution relative to simpler backoff statistics. Similarly, Ariya Rastrow et al. applied information-theoretic criteria to structured language models in Efficient Structured Language Modeling for Speech Recognition, proving that context nodes with minimal impact on the global distribution could be safely removed. The authors eliminated nearly 80% of the model's parameters with negligible loss in accuracy.
This framework scales seamlessly to massive, web-scale production systems. For example, in the domain of sponsored search advertising, Thore Graepel demonstrated in Web-Scale Bayesian Click-Through rate Prediction for Sponsored Search Advertising in Microsoft's Bing Search Engine that features can be aggressively pruned if the KL divergence between their current posterior distribution and the prior distribution falls below a strict threshold. If a feature's posterior hasn't moved significantly from the prior, it provides little to no information gain and can be discarded to save memory.
The same mechanism applies to data purification. When dealing with ambiguous or noisy datasets, Yangfan Liu et al. showed in Learning with Partial-Label and Unlabeled Data: A Uniform Treatment for Supervision Redundancy and Insufficiency that candidate label sets can be purified by calculating the KL divergence between the model's output distribution with and without a specific candidate label, systematically removing labels that cause the minimum information loss.
Entropy as a Proxy for Redundancy
While calculating full distributional divergence across an entire dataset is mathematically rigorous, it can be computationally expensive. As a result, researchers often look at the entropy or information density of individual components as a localized proxy for redundancy.
In deep neural networks, layers or filters that exhibit low information entropy contain highly redundant or uniform representations, making them prime targets for compression. Yiheng Lu et al. formalized this in Entropy Induced Pruning Framework for Convolutional Neural Networks by introducing a metric that quantifies filter redundancy through the entropy of weight matrix eigenvalues. By assigning higher pruning rates to layers with lower entropy scores, the authors focused compression efforts precisely where information was most redundant. For edge and mobile applications, Zhenyu Wang et al. utilized a similar strategy in SAS: Painting Detection and Recognition via Smart Art System With Mobile Devices, calculating the entropy of individual convolutional filters to discard those with low information density, drastically reducing latency on hardware-constrained devices.
Conversely, in stochastic architectures, entropy can be used to identify and prune components that introduce too much uninformative noise. Tianyuan Yu et al. demonstrated in Simple and Effective Stochastic Neural Networks that stochastic neural networks can achieve extreme structural compression by pruning neurons that maintain high entropy across the training set, effectively cleaning up the model's internal uncertainty.
In feature-heavy discriminative language models, evaluating the exact change in model entropy caused by removing specific feature-target pairs yields massive efficiency gains. Barlas Oguz et al. showed in Entropy Based Pruning for Non-Negative Matrix Based Language Models with Contextual Features that this entropy-change strategy vastly outperforms simple heuristics like mutual information, achieving a 60-fold parameter reduction while fully maintaining the performance of high-dimensional contextual models.
This balance between predictive value and entropy cost can even be built directly into a model's structural optimization. Matthew Brand developed a trimming criterion in An Entropic Estimator for Structure Discovery that compares a parameter's contribution to log-likelihood against its contribution to total model entropy. If a parameter's entropy cost outweighs its predictive value, it is driven to zero, automatically discovering a sparse and interpretable model architecture.
Compressing Inputs and Contexts via Surprisal
The core intuition of information-theoretic pruning extends beyond model weights to the data inputs themselves. In modern autoregressive large language models, long context windows create significant computational bottlenecks. Just as we can prune redundant parameters from a model, we can prune redundant tokens from a prompt by measuring their surprisal—the negative log-likelihood of a token given its preceding context.
If a token has low surprisal, it means the model highly anticipated its appearance; it is predictable and therefore carries low self-information. Yucheng Li et al. introduced this concept to prompt engineering in Compressing Context to Enhance Inference Efficiency of Large Language Models. By using a small, efficient language model to calculate the self-information of prompt segments, their "Selective Context" method filters out highly predictable, redundant text. This reduces inference memory and latency by over 30% while keeping reasoning capabilities intact.
Input Prompt: "The weather is very nice and sunny today."
Per-token Surprisal: [ High | Low | Low | Low | High | High ]
Pruned Prompt: "Weather nice sunny today."
As detailed in a comprehensive survey by Zongqian Li et al., Prompt Compression for Large Language Models: A Survey, discrete prompt compression relies heavily on calculating the information value or perplexity of individual tokens. Removing tokens that fall below an informational threshold allows long instructions to be compressed into dense, high-signal inputs, bypassing context-window bottlenecks without altering the underlying model weights.
Alternative Divergence Metrics
While KL divergence and Shannon entropy dominate the literature, specific problem domains benefit from alternative information-theoretic metrics like the Hellinger distance or Jensen-Shannon divergence, which offer distinct mathematical advantages.
For instance, in online learning scenarios where data arrives in an infinite stream, models like Gaussian Processes suffer from a "sample complexity bottleneck" where memory requirements grow cubically with the number of samples. To solve this, Hrusikesha Pradhan et al. developed Parsimonious Online Gaussian Processes in Consistent online Gaussian process regression without the sample complexity bottleneck. Their framework uses the Hellinger distance to measure the divergence between the full posterior distribution and a reduced-size approximation. Because the Hellinger distance is a true metric bounded between 0 and 1, it allows the system to enforce a strict, predictable "error budget," discarding historical data points only when the approximation remains safely within the budget.
In spatial and semantic mapping for robotics, symmetric metrics like the Jensen-Shannon divergence are preferred. When a robot builds a 3D semantic map, storing every spatial voxel consumes prohibitive amounts of memory. Daniel T. Larsson et al. addressed this in Information-theoretic Abstraction of Semantic Octree Models for Integrated Perception and Planning by calculating the Jensen-Shannon divergence between parent and child nodes in a spatial tree. If the semantic probability distributions of adjacent spatial regions are highly similar (resulting in low divergence), the nodes are merged and pruned. This ensures that the robot maintains high-resolution detail only for task-relevant boundaries while aggressively compressing uniform spaces like the sky or open floors.
Whether applied to the temporal steps of spiking neural networks via Taylor-expanded KL approximations (Shuiping Gou et al. in Dynamic spatio-temporal pruning for efficient spiking neural networks), or to active learning data selection (Shlomo Argamon et al. in Minimizing Manual Annotation Cost in Supervised Training from Corpora), the core intuition remains identical. By shifting our perspective from how large a parameter or data point is to how much information it contributes to the overall distribution, we can build lean, highly efficient models that retain the intelligence of their over-parameterized ancestors.
Key Foundational Work
Truncation on Combined Word-Based and Class-Based Language Model Using Kullback-Leibler Distance Criterion (1997) — contributed 2 concept elements to this direction.
Minimizing Manual Annotation Cost in Supervised Training from Corpora (1996) — contributed 1 concept elements to this direction.