Embedding Compression for Text Classification Using Dictionary Screening

Hansheng Wang, Jing Zhou, Xinru Jing, Muyu Liu, Hansheng Wang 0002arXiv 2022 · DOI · arXiv

Abstract

As Natural Language Processing (NLP) models grow in complexity, their deployment on resource-constrained edge devices becomes increasingly challenging. While existing compression techniques often target neural network weights or architectures, the embedding layer—which maps vocabulary to high-dimensional vectors—remains a significant source of parameter redundancy. This paper explores "Dictionary Screening," a task-specific compression method that identifies and retains only the most influential keywords for a given classification task. By utilizing a Class Probability Estimator (CPE), the proposed method reduces model parameters by over 90% while maintaining, and occasionally improving, classification accuracy.

Introduction

In the current NLP landscape, the pursuit of higher accuracy often leads to larger models. However, for deployment on smartphones, tablets, and IoT devices, memory and battery life are critical constraints. Traditional compression methods, such as pruning or quantization, typically focus on the hidden layers of Recurrent Neural Networks (RNNs) or Convolutional Neural Networks (CNNs). Yet, the embedding layer often accounts for a disproportionate share of the total parameter count. For instance, a dictionary of 100,000 words mapped to 128-dimensional vectors requires 12.8 million parameters before any computation even occurs.

This research addresses the redundancy within the input dictionary itself. By screening the vocabulary to keep only task-relevant words, the authors demonstrate that it is possible to slash model size significantly without sacrificing performance.

Methodology: Dictionary Screening via CPE

The core of the proposed approach is the Class Probability Estimator (CPE) screening method. Unlike frequency-based methods like TF-IDF, CPE is task-aware, evaluating a word's importance based on its actual impact on the model's decision-making process. The workflow consists of four primary stages:

  1. Benchmark Training: A standard model (e.g., TextCNN or BiLSTM) is trained on the full, uncompressed dictionary to establish a performance baseline.
  2. Keyword Importance Evaluation: For every word in the dictionary, the researchers measure how the model's predicted class probability changes when that word is removed from the training documents. If the $\ell_2$-distance between the original and modified probability distributions is high, the word is considered highly influential. If the change is negligible, the word is deemed redundant noise.
  3. Screening: Words are ranked by their importance scores. Only the top $K$ keywords (typically ranging from 1,000 to 5,000) are retained.
  4. Re-training: The text sequences are reconstructed using only the screened dictionary, and a new, significantly smaller model is trained from scratch.

Experimental Setup

The effectiveness of Dictionary Screening was evaluated using two popular architectures—TextCNN and TextBiLSTM—across four large-scale datasets:

  • AG’s News: English news classification (4 classes).
  • Sougou News: Chinese news classification (using characters to increase vocabulary complexity).
  • DBPedia: Entity classification (14 classes).
  • Amazon Review Polarity (ARP): Large-scale sentiment analysis (2 classes).

Key Results and Discussion

Massive Parameter Reduction

The empirical results demonstrate that Dictionary Screening is highly effective at reducing model footprints. On the AG’s News dataset, the method achieved a 95.24% reduction in parameters. Despite this aggressive compression, the model achieved an accuracy of 89.43%, which was slightly higher than the 89.37% achieved by the full-scale benchmark model.

The Regularization Effect

One of the most significant findings is that removing "unimportant" words can act as a form of regularization. By stripping away noise—words that do not contribute to the specific classification task—the model is less likely to overfit on irrelevant features. This explains why compressed models occasionally outperform their larger counterparts.

Computational Efficiency

Beyond storage savings, Dictionary Screening naturally shortens the average text sequence length. Because the model ignores non-keywords during inference, there are fewer tokens to process per document. This leads to faster inference times, further benefiting edge device deployment.

Comparison with Baselines

CPE consistently outperformed traditional statistical methods such as TF-IDF and t-statistic-based screening. While TF-IDF identifies frequent words, it fails to capture the semantic necessity of words for specific labels, a gap that the task-aware CPE method successfully bridges.

Conclusion and Future Work

Dictionary Screening offers a powerful alternative to traditional model compression. By focusing on the vocabulary rather than just the network weights, researchers can achieve parameter reductions of over 90% with negligible performance loss.

Future research directions include automating the selection of the optimal keyword count ($K$) and testing the method on more complex generative tasks, such as machine translation, where the preservation of grammatical "noise" words may be more critical than in pure classification tasks. Additionally, expanding the study to morphologically rich languages will provide further insight into the universal applicability of keyword-based screening.

Summary updated

Abstract

In this paper, we propose a dictionary screening method for embedding compression in text classification tasks. The key purpose of this method is to evaluate the importance of each keyword in the dictionary. To this end, we first train a pre-specified recurrent neural network-based model using a full dictionary. This leads to a benchmark model, which we then use to obtain the predicted class probabilities for each sample in a dataset. Next, to evaluate the impact of each keyword in affecting the predicted class probabilities, we develop a novel method for assessing the importance of each keyword in a dictionary. Consequently, each keyword can be screened, and only the most important keywords are reserved. With these screened keywords, a new dictionary with a considerably reduced size can be constructed. Accordingly, the original text sequence can be substantially compressed. The proposed method leads to significant reductions in terms of parameters, average text sequence, and dictionary size. Meanwhile, the prediction power remains very competitive compared to the benchmark model. Extensive numerical studies are presented to demonstrate the empirical performance of the proposed method.
BibTeX
@misc{wang2022embedding,
  author = {Hansheng Wang and Jing Zhou and Xinru Jing and Muyu Liu and Hansheng Wang},
  title = {{Embedding Compression for Text Classification Using Dictionary Screening}},
  year = {2022},
  eprint = {2211.12715},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.12715},
}
Figure