Subword-based Compact Reconstruction of Word Embeddings

Shota Sasaki, Jun Suzuki, Kentaro InuiNAACL 2019 · OpenReview

Abstract

High-quality word embeddings like GloVe and fastText are foundational to modern NLP but suffer from massive memory requirements and an inability to handle out-of-vocabulary (OOV) words. This paper introduces a reconstruction framework that compresses these embeddings using memory-shared hashing and a Key-Value-Query (KVQ) self-attention mechanism. The proposed method achieves up to a 10x reduction in memory usage with negligible performance loss while significantly improving OOV handling.

Introduction

Standard word embeddings present a significant trade-off in practical applications. High-quality sets such as fastText.600B require over 2GB of VRAM, a prohibitive cost for resource-constrained environments or multi-model deployments. Additionally, word-based models are limited by a fixed vocabulary; any word not seen during training is treated as a generic "unknown" token, stripping away semantic nuance. While subword-based models mitigate OOV issues, they often require extensive storage for their own subword dictionaries. This research proposes a method to distill the knowledge of massive pre-trained embeddings into a compact, subword-based representation.

Methodology

The authors propose representing each word embedding as a weighted combination of its constituent subwords (character n-grams). Two primary innovations drive the efficiency and accuracy of this reconstruction:

Memory-Shared Embeddings (Hashing)

To minimize the parameter count, the model avoids dedicated vectors for every unique subword. Instead, a hash function maps subwords to a fixed, smaller pool of $H$ vectors. This surjective mapping allows multiple subwords to share physical memory slots, drastically reducing the total storage footprint.

KVQ Self-Attention Mechanism

Simple summation of hashed vectors is often insufficient due to collisions. To resolve this, the authors implement a variant of the Key-Value-Query (KVQ) self-attention mechanism.

The KVQ self-attention mechanism for subword-based word reconstruction.

As illustrated in Figure 2, the model treats subword vectors as keys, values, and queries to calculate context-dependent weights. This allows the framework to dynamically emphasize the most semantically relevant subwords for a given word while filtering out the "noise" inherent in shared hashed vectors.

Experimental Evaluation

The researchers evaluated the framework across word similarity datasets (e.g., MEN, Rare Word), analogy tasks (Google Analogy), and downstream applications including Named Entity Recognition (CoNLL-2003) and Textual Entailment (SNLI).

Key Results

The KVQ-based approach demonstrated a superior balance between compression and performance compared to baseline summation methods.

Performance comparison on word similarity (Spearman Correlation) and analogy (Accuracy) tasks across different subword embedding counts.

As shown in Figure 3, the KVQ-H and KVQ-FH models consistently outperform summation baselines (SUM-H, SUM-F). Key findings include:

  • Significant Compression: The KVQ-H model reduced the size of fastText.600B from 2.2GB to 0.59GB (nearly 4x) while maintaining or slightly improving analogy task performance.
  • Extreme Efficiency: Even at a 10x reduction (0.23GB), the performance degradation on downstream tasks like NER was less than 1.0 point.
  • OOV Robustness: In synthetic OOV tests, the KVQ-FH method outperformed existing state-of-the-art reconstruction methods by 2 to 6 points.

Conclusion

This paper demonstrates that word embedding reconstruction can serve as an effective post-processing step to "distill" large models into efficient, production-ready versions. By combining hashing with self-attention, the proposed framework provides a robust solution for deploying high-performance NLP models in memory-limited environments without sacrificing semantic depth or OOV handling capabilities.

Summary updated

BibTeX
@inproceedings{sasaki2019subword,
  author = {Shota Sasaki and Jun Suzuki and Kentaro Inui},
  title = {{Subword-based Compact Reconstruction of Word Embeddings}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=SJbN2fZuWB},
}
Figure