Abstract
Compressing word embeddings is essential for deploying NLP models in memory-constrained environments. However, existing measures of compression quality often fail to distinguish between embeddings that perform well on downstream tasks and those that do not. We propose the eigenspace overlap score as a new measure and relate it to downstream performance through generalization bounds in linear and logistic regression. We demonstrate that a simple uniform quantization method effectively preserves this score, explaining its strong empirical performance. Finally, we show that using the eigenspace overlap score as a selection criterion identifies better-performing embeddings with up to $2\times$ lower error rates than existing metrics.
The Challenge of Compression Metrics
Word embeddings like GloVe or fastText are fundamental to modern NLP but require significant memory. To deploy these models on edge devices, compression is necessary. However, the primary challenge is not just reducing size, but ensuring the compressed embeddings remain effective for downstream tasks.
Traditionally, researchers have relied on metrics such as reconstruction error or PIP loss to evaluate compression quality. Unfortunately, these metrics are often poor predictors of actual task performance. As shown in Figure 1, an embedding might appear mathematically accurate according to PIP loss but fail significantly on tasks like question answering (SQuAD). Notably, simple uniform quantization can outperform complex dimensionality reduction even at much higher compression rates (e.g., 32x vs 6x), a fact that traditional metrics fail to capture.
The Eigenspace Overlap Score
To address this gap, we introduce the eigenspace overlap score. Unlike metrics that focus on individual word vector displacements, this score measures how well the subspace spanned by the eigenvectors of the compressed embedding's Gram matrix aligns with the original. This approach focuses on preserving the geometric "skeleton" of the data rather than exact point positions.
Our theoretical analysis links this score to generalization. We derive bounds for linear and logistic regression, proving that a higher eigenspace overlap score directly corresponds to lower expected loss on unseen data. This suggests that for downstream tasks, preserving the principal directions of the data is more critical than minimizing local noise.
Key Insights: The Power of Subspace Alignment
Our research yielded several surprising insights:
- Uniform Quantization is Highly Effective: A simple uniform quantization method—scaling values and rounding—consistently matches or outperforms complex deep learning-based methods like Deep Compositional Code Learning (DCCL). We provide a theoretical lower bound for the overlap score under uniform quantization, explaining its robustness.
- Subspaces Matter More Than Vectors: Downstream models can often learn to compensate for the noise introduced by quantization as long as the overall orientation of the subspace is preserved.
- Superior Predictive Power: The eigenspace overlap score is a remarkably robust predictor of performance. While existing metrics like PIP loss showed weak Spearman correlation ($\rho \approx 0.3–0.4$), our score consistently achieved $\rho$ values between 0.81 and 0.93.
Empirical Results and Selection Accuracy
We validated our findings across diverse embeddings (GloVe, fastText, BERT) and tasks (SQuAD, SST-1, GLUE).
As illustrated in Figure 4, the eigenspace overlap measure (represented as $1-\mathcal{E}$) provides a much clearer signal for downstream performance than PIP loss or other distance-based metrics. By using this score as a selection criterion, we achieved up to 2× lower selection error rates when choosing the best compressed embedding from a candidate pool, avoiding the prohibitive cost of training full models for every task.
Conclusion and Future Work
The eigenspace overlap score provides a theoretically grounded and empirically robust metric for evaluating compressed word embeddings. While our current theoretical proofs are focused on linear models, the empirical success across complex neural architectures suggests broader applicability. Future work will explore extending these generalization bounds to deep non-linear models and applying the overlap score to other domains, such as graph embeddings or internal neural network activations.
Summary updated