On the Downstream Performance of Compressed Word Embeddings

Avner May, Jian Zhang, Tri Dao, Christopher Ré

Abstract

Compressing word embeddings is essential for deploying NLP models in memory-constrained environments. However, existing measures of compression quality often fail to distinguish between embeddings that perform well on downstream tasks and those that do not. We propose the eigenspace overlap score as a new measure and relate it to downstream performance through generalization bounds in linear and logistic regression. We demonstrate that a simple uniform quantization method effectively preserves this score, explaining its strong empirical performance. Finally, we show that using the eigenspace overlap score as a selection criterion identifies better-performing embeddings with up to $2\times$ lower error rates than existing metrics.

The Challenge of Compression Metrics

Word embeddings like GloVe or fastText are fundamental to modern NLP but require significant memory. To deploy these models on edge devices, compression is necessary. However, the primary challenge is not just reducing size, but ensuring the compressed embeddings remain effective for downstream tasks.

Traditionally, researchers have relied on metrics such as reconstruction error or PIP loss to evaluate compression quality. Unfortunately, these metrics are often poor predictors of actual task performance. As shown in Figure 1, an embedding might appear mathematically accurate according to PIP loss but fail significantly on tasks like question answering (SQuAD). Notably, simple uniform quantization can outperform complex dimensionality reduction even at much higher compression rates (e.g., 32x vs 6x), a fact that traditional metrics fail to capture.

Figure 1: F1 score vs. PIP loss for GloVe on SQuAD. Traditional metrics like PIP loss show weak correlation with downstream performance, often failing to distinguish between effective and ineffective compression methods.

The Eigenspace Overlap Score

To address this gap, we introduce the eigenspace overlap score. Unlike metrics that focus on individual word vector displacements, this score measures how well the subspace spanned by the eigenvectors of the compressed embedding's Gram matrix aligns with the original. This approach focuses on preserving the geometric "skeleton" of the data rather than exact point positions.

Our theoretical analysis links this score to generalization. We derive bounds for linear and logistic regression, proving that a higher eigenspace overlap score directly corresponds to lower expected loss on unseen data. This suggests that for downstream tasks, preserving the principal directions of the data is more critical than minimizing local noise.

Key Insights: The Power of Subspace Alignment

Our research yielded several surprising insights:

  1. Uniform Quantization is Highly Effective: A simple uniform quantization method—scaling values and rounding—consistently matches or outperforms complex deep learning-based methods like Deep Compositional Code Learning (DCCL). We provide a theoretical lower bound for the overlap score under uniform quantization, explaining its robustness.
  2. Subspaces Matter More Than Vectors: Downstream models can often learn to compensate for the noise introduced by quantization as long as the overall orientation of the subspace is preserved.
  3. Superior Predictive Power: The eigenspace overlap score is a remarkably robust predictor of performance. While existing metrics like PIP loss showed weak Spearman correlation ($\rho \approx 0.3–0.4$), our score consistently achieved $\rho$ values between 0.81 and 0.93.

Empirical Results and Selection Accuracy

We validated our findings across diverse embeddings (GloVe, fastText, BERT) and tasks (SQuAD, SST-1, GLUE).

Figure 4: Comparison of various metrics (PIP loss, $\Delta_{max}$, $\Delta$, and the proposed $1-\mathcal{E}$ score) against downstream F1 score. The eigenspace overlap measure (rightmost) demonstrates a significantly stronger correlation with task performance than traditional measures.

As illustrated in Figure 4, the eigenspace overlap measure (represented as $1-\mathcal{E}$) provides a much clearer signal for downstream performance than PIP loss or other distance-based metrics. By using this score as a selection criterion, we achieved up to 2× lower selection error rates when choosing the best compressed embedding from a candidate pool, avoiding the prohibitive cost of training full models for every task.

Conclusion and Future Work

The eigenspace overlap score provides a theoretically grounded and empirically robust metric for evaluating compressed word embeddings. While our current theoretical proofs are focused on linear models, the empirical success across complex neural architectures suggests broader applicability. Future work will explore extending these generalization bounds to deep non-linear models and applying the overlap score to other domains, such as graph embeddings or internal neural network activations.

Summary updated

Abstract

Compressing word embeddings is important for deploying NLP models in memory-constrained settings. However, understanding what makes compressed embeddings perform well on downstream tasks is challenging---existing measures of compression quality often fail to distinguish between embeddings that perform well and those that do not. We thus propose the eigenspace overlap score as a new measure. We relate the eigenspace overlap score to downstream performance by developing generalization bounds for the compressed embeddings in terms of this score, in the context of linear and logistic regression. We then show that we can lower bound the eigenspace overlap score for a simple uniform quantization compression method, helping to explain the strong empirical performance of this method. Finally, we show that by using the eigenspace overlap score as a selection criterion between embeddings drawn from a representative set we compressed, we can efficiently identify the better performing embedding with up to $2\times$ lower selection error rates than the next best measure of compression quality, and avoid the cost of training a model for each task of interest.
BibTeX
@inproceedings{may2019downstream,
  author = {Avner May and Jian Zhang and Tri Dao and Christopher Ré},
  title = {{On the Downstream Performance of Compressed Word Embeddings}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=QWk-AsmmGda},
}
Figure