On the Downstream Performance of Compressed Word Embeddings

Avner May, Jian Zhang, Tri Dao, Christopher Ré

Word embeddings like GloVe and fastText are foundational to modern Natural Language Processing (NLP). However, their massive memory footprints make deployment on mobile devices or cost-constrained data centers challenging. While numerous compression techniques exist—from simple K-means clustering to complex deep learning approaches—researchers have lacked a reliable metric to evaluate "compression quality" before deploying models.

Existing metrics, such as reconstruction error or Pairwise Inner Product (PIP) loss, often fail to correlate with actual downstream task performance. A compressed embedding might appear "noisy" according to standard metrics but still yield high accuracy on tasks like question answering. This paper bridges the gap between compression theory and practical downstream performance by introducing a novel metric: the eigenspace overlap score.

The Eigenspace Overlap Score

Instead of measuring how much individual numerical values change after compression, the eigenspace overlap score ($E$) evaluates the geometric orientation of the data. It measures the degree of overlap between the subspaces spanned by the eigenvectors of the original and compressed embedding matrices.

The authors provide mathematical proofs (specifically for linear and logistic regression) demonstrating that a higher eigenspace overlap score directly leads to better expected performance on new data. By focusing on "average-case" performance across typical tasks rather than "worst-case" spectral measures, this score becomes a highly reliable predictor of downstream success.

The Power of Simplicity: Uniform Quantization

One of the paper's most surprising insights is that uniform quantization—a relatively simple technique that rounds values to a fixed set of levels—consistently matches or outperforms more complex, state-of-the-art methods like Deep Compositional Code Learning (DCCL). The authors prove theoretically that uniform quantization naturally maintains a high eigenspace overlap score, even at aggressive compression rates.

Performance of compression methods on SQuAD relative to PIP loss. Uniform quantization maintains high F1 scores even at 32x compression, highlighting the disconnect between PIP loss and downstream performance.

Empirical Results

The researchers evaluated their findings across diverse NLP benchmarks, including SQuAD (reading comprehension), SST-1 (sentiment analysis), and GLUE tasks (MNLI and QQP), using GloVe, fastText, and BERT WordPiece embeddings.

  • Superior Correlation: The eigenspace overlap score demonstrated a significantly higher Spearman correlation with downstream performance (up to 0.93) compared to previous metrics like PIP loss (which hovered around 0.34–0.49).
  • Efficient Selection: Using this score as a "litmus test" to select the best compressed embedding from a set of candidates resulted in up to 2× lower selection error rates than the next best measure. This avoids the computational cost of training a full model for every task just to evaluate an embedding.
  • Validation of Uniform Quantization: In tests on SQuAD, uniform quantization at a 32× compression rate achieved an F1 score only 0.47 points below the uncompressed baseline, significantly outperforming more complex methods.
Correlation between the eigenspace-based metric and F1 score on SQuAD. The new metric achieves a strong correlation ($\rho = -0.91$), vastly outperforming traditional metrics.

Limitations and Future Work

While empirical results show the score works well for deep neural networks, the current theoretical generalization bounds are limited to linear and logistic regression. Extending these mathematical proofs to non-linear models remains an open challenge. Furthermore, the study primarily examines scenarios where embeddings are "frozen" during downstream training; investigating how the score relates to performance when embeddings are fine-tuned is a promising direction. Finally, future work could extend this metric beyond word embeddings to graph embeddings or internal neural network activations.

Summary updated

Abstract

Compressing word embeddings is important for deploying NLP models in memory-constrained settings. However, understanding what makes compressed embeddings perform well on downstream tasks is challenging---existing measures of compression quality often fail to distinguish between embeddings that perform well and those that do not. We thus propose the eigenspace overlap score as a new measure. We relate the eigenspace overlap score to downstream performance by developing generalization bounds for the compressed embeddings in terms of this score, in the context of linear and logistic regression. We then show that we can lower bound the eigenspace overlap score for a simple uniform quantization compression method, helping to explain the strong empirical performance of this method. Finally, we show that by using the eigenspace overlap score as a selection criterion between embeddings drawn from a representative set we compressed, we can efficiently identify the better performing embedding with up to 2x lower selection error rates than the next best measure of compression quality, and avoid the cost of training a separate model for each task of interest.
BibTeX
@inproceedings{may2019downstream,
  author = {Avner May and Jian Zhang and Tri Dao and Christopher Ré},
  title = {{On the Downstream Performance of Compressed Word Embeddings}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=QWk-AsmmGda},
}
Figure