Improving Language Model Size Reduction using Better Pruning Criteria

Jianfeng Gao, Min ZhangACL 2002 · OpenReview

Abstract

Reducing language model (LM) size is a critical issue when applying a LM to realistic applications which have memory constraints. In this paper, three measures are studied for the purpose of LM pruning: probability, rank, and entropy. We evaluated the performance of these three pruning criteria in a real application of Chinese text input in terms of character error rate (CER). We first present an empirical comparison, showing that rank performs the best in most cases. We also show that the high-performance of rank lies in its strong correlation with error rate. We then present a novel method of combining two criteria in model pruning. Experimental results show that the combined criterion consistently leads to smaller models than the models pruned using either of the criteria separately, at the same CER.

Introduction

Language models are the backbone of technologies like speech recognition and text input systems, but they are notoriously memory-hungry. While we can train massive models on billions of words, realistic applications—especially on mobile devices—have strict memory limits. The challenge is pruning: removing less important parameters (n-grams) from the model to save space. Historically, researchers used "count cutoffs" or "probability-based" criteria. However, these metrics don't always correlate well with actual task performance, such as the character error rate (CER). This paper explores pruning criteria that better estimate the actual performance loss of a pruned model.

Methodology

Pruning Criteria

The authors evaluate three distinct pruning measures:

  1. Probability (Baseline): Prunes bigrams that have the least impact on the model's perplexity.
  2. Rank: Focuses on the position of a word in an ordered list of predictions. If pruning a bigram doesn't significantly change the "rank" of the correct word, it is removed.
  3. Entropy: Measures how much the uncertainty of the entire probability distribution changes when a bigram is removed.

Combining Criteria

A novel contribution of this work is the combination of criteria. Instead of picking just one measure, the authors developed a modified thresholding algorithm that uses two criteria simultaneously (e.g., Rank + Entropy). To make this efficient, they used a linear regression model to find the optimal function relating the two pruning thresholds, allowing them to find the best-performing model size for any given error rate.

Experimental Setup

The researchers focused on a Chinese text input application (converting Pinyin to characters). The training data consisted of a balanced corpus of approximately 26 million Chinese characters, while the test data comprised a proofread set of 0.5 million characters. The experiments utilized character-based bigram models.

Results and Analysis

Individual Performance

The study found that Rank is a significantly better proxy for error rate than Perplexity. In a decoder, the system is trying to pick the top candidate; if a word moves from rank 100 to 101, it doesn't matter, but moving from rank 1 to 2 is critical. Rank-based pruning captures this intuition, correlating more strongly with the Character Error Rate (CER).

Comparison of individual pruning criteria: rank, probability, entropy, and count cutoff.

As shown in Figure 1, the rank-based criterion consistently achieves a lower average error rate for the same number of bigrams compared to probability, entropy, or simple count cutoffs. Using Rank alone produced models that were 55-85% the size of the baseline at the same error rate.

The Power of Complementarity

The authors discovered that different criteria "disagree" on which bigrams are useless. For example, the overlap between bigrams pruned by Rank vs. Entropy was only about 63.6%. By combining them, the system keeps the bigrams deemed important by either measure, leading to a more robust "distilled" model.

Performance of combined pruning criteria compared to individual ones.

As illustrated in Figure 3, the combination of Rank + Entropy was the clear winner. For instance, to achieve a CER of 13.8%, the standard Probability model required 8.2 MB, while the Rank + Entropy model required only 4.25 MB—a 48.1% total size reduction over the baseline.

Conclusion

This research demonstrates that moving beyond standard probability-based pruning to use rank and entropy—and specifically combining them—can significantly reduce model sizes without sacrificing accuracy. While modern LMs have shifted toward neural architectures, the core philosophy of finding pruning criteria that correlate with downstream task performance remains a central theme in model compression and distillation.

Summary updated

Abstract

Reducing language model (LM) size is a critical issue when applying a LM to realistic applications which have memory constraints. In this paper, three measures are studied for the purpose of LM pruning. They are probability, rank, and entropy. We evaluated the performance of the three pruning criteria in a real application of Chinese text input in terms of character error rate (CER). We first present an empirical comparison, showing that rank performs the best in most cases. We also show that the high-performance of rank lies in its strong correlation with error rate. We then present a novel method of combining two criteria in model pruning. Experimental results show that the combined criterion consistently leads to smaller models than the models pruned using either of the criteria separately, at the same CER.
BibTeX
@inproceedings{gao2002improving,
  author = {Jianfeng Gao and Min Zhang},
  title = {{Improving Language Model Size Reduction using Better Pruning Criteria}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2002},
  url = {https://openreview.net/forum?id=r1VzFngu-B},
}
Figure