Abstract
High-quality word embeddings like GloVe and fastText are foundational to modern NLP but suffer from massive memory requirements and an inability to handle out-of-vocabulary (OOV) words. This paper introduces a reconstruction framework that compresses these embeddings using memory-shared hashing and a Key-Value-Query (KVQ) self-attention mechanism. The proposed method achieves up to a 10x reduction in memory usage with negligible performance loss while significantly improving OOV handling.
Introduction
Standard word embeddings present a significant trade-off in practical applications. High-quality sets such as fastText.600B require over 2GB of VRAM, a prohibitive cost for resource-constrained environments or multi-model deployments. Additionally, word-based models are limited by a fixed vocabulary; any word not seen during training is treated as a generic "unknown" token, stripping away semantic nuance. While subword-based models mitigate OOV issues, they often require extensive storage for their own subword dictionaries. This research proposes a method to distill the knowledge of massive pre-trained embeddings into a compact, subword-based representation.
Methodology
The authors propose representing each word embedding as a weighted combination of its constituent subwords (character n-grams). Two primary innovations drive the efficiency and accuracy of this reconstruction:
Memory-Shared Embeddings (Hashing)
To minimize the parameter count, the model avoids dedicated vectors for every unique subword. Instead, a hash function maps subwords to a fixed, smaller pool of $H$ vectors. This surjective mapping allows multiple subwords to share physical memory slots, drastically reducing the total storage footprint.
KVQ Self-Attention Mechanism
Simple summation of hashed vectors is often insufficient due to collisions. To resolve this, the authors implement a variant of the Key-Value-Query (KVQ) self-attention mechanism.
As illustrated in Figure 2, the model treats subword vectors as keys, values, and queries to calculate context-dependent weights. This allows the framework to dynamically emphasize the most semantically relevant subwords for a given word while filtering out the "noise" inherent in shared hashed vectors.
Experimental Evaluation
The researchers evaluated the framework across word similarity datasets (e.g., MEN, Rare Word), analogy tasks (Google Analogy), and downstream applications including Named Entity Recognition (CoNLL-2003) and Textual Entailment (SNLI).
Key Results
The KVQ-based approach demonstrated a superior balance between compression and performance compared to baseline summation methods.
As shown in Figure 3, the KVQ-H and KVQ-FH models consistently outperform summation baselines (SUM-H, SUM-F). Key findings include:
- Significant Compression: The KVQ-H model reduced the size of fastText.600B from 2.2GB to 0.59GB (nearly 4x) while maintaining or slightly improving analogy task performance.
- Extreme Efficiency: Even at a 10x reduction (0.23GB), the performance degradation on downstream tasks like NER was less than 1.0 point.
- OOV Robustness: In synthetic OOV tests, the KVQ-FH method outperformed existing state-of-the-art reconstruction methods by 2 to 6 points.
Conclusion
This paper demonstrates that word embedding reconstruction can serve as an effective post-processing step to "distill" large models into efficient, production-ready versions. By combining hashing with self-attention, the proposed framework provides a robust solution for deploying high-performance NLP models in memory-limited environments without sacrificing semantic depth or OOV handling capabilities.
Summary updated