Word embeddings like GloVe and fastText are foundational to modern Natural Language Processing (NLP). However, their massive memory footprints make deployment on mobile devices or cost-constrained data centers challenging. While numerous compression techniques exist—from simple K-means clustering to complex deep learning approaches—researchers have lacked a reliable metric to evaluate "compression quality" before deploying models.
Existing metrics, such as reconstruction error or Pairwise Inner Product (PIP) loss, often fail to correlate with actual downstream task performance. A compressed embedding might appear "noisy" according to standard metrics but still yield high accuracy on tasks like question answering. This paper bridges the gap between compression theory and practical downstream performance by introducing a novel metric: the eigenspace overlap score.
The Eigenspace Overlap Score
Instead of measuring how much individual numerical values change after compression, the eigenspace overlap score ($E$) evaluates the geometric orientation of the data. It measures the degree of overlap between the subspaces spanned by the eigenvectors of the original and compressed embedding matrices.
The authors provide mathematical proofs (specifically for linear and logistic regression) demonstrating that a higher eigenspace overlap score directly leads to better expected performance on new data. By focusing on "average-case" performance across typical tasks rather than "worst-case" spectral measures, this score becomes a highly reliable predictor of downstream success.
The Power of Simplicity: Uniform Quantization
One of the paper's most surprising insights is that uniform quantization—a relatively simple technique that rounds values to a fixed set of levels—consistently matches or outperforms more complex, state-of-the-art methods like Deep Compositional Code Learning (DCCL). The authors prove theoretically that uniform quantization naturally maintains a high eigenspace overlap score, even at aggressive compression rates.
Empirical Results
The researchers evaluated their findings across diverse NLP benchmarks, including SQuAD (reading comprehension), SST-1 (sentiment analysis), and GLUE tasks (MNLI and QQP), using GloVe, fastText, and BERT WordPiece embeddings.
- Superior Correlation: The eigenspace overlap score demonstrated a significantly higher Spearman correlation with downstream performance (up to 0.93) compared to previous metrics like PIP loss (which hovered around 0.34–0.49).
- Efficient Selection: Using this score as a "litmus test" to select the best compressed embedding from a set of candidates resulted in up to 2× lower selection error rates than the next best measure. This avoids the computational cost of training a full model for every task just to evaluate an embedding.
- Validation of Uniform Quantization: In tests on SQuAD, uniform quantization at a 32× compression rate achieved an F1 score only 0.47 points below the uncompressed baseline, significantly outperforming more complex methods.
Limitations and Future Work
While empirical results show the score works well for deep neural networks, the current theoretical generalization bounds are limited to linear and logistic regression. Extending these mathematical proofs to non-linear models remains an open challenge. Furthermore, the study primarily examines scenarios where embeddings are "frozen" during downstream training; investigating how the score relates to performance when embeddings are fine-tuned is a promising direction. Finally, future work could extend this metric beyond word embeddings to graph embeddings or internal neural network activations.
Summary updated