Abstract
In neural network-based models for natural language processing (NLP), the largest part of the parameters often consists of word embeddings. Conventional models prepare a large embedding matrix whose size depends on the vocabulary size. Therefore, storing these models in memory and disk storage is costly. In this study, to reduce the total number of parameters, the embeddings for all words are represented by transforming a shared embedding. The proposed method, ALONE (all word embeddings from one), constructs the embedding of a word by modifying the shared embedding with a filter vector, which is word-specific but non-trainable. Then, we input the constructed embedding into a feed-forward neural network to increase its expressiveness. Naively, the filter vectors occupy the same memory size as the conventional embedding matrix, which depends on the vocabulary size. To solve this issue, we also introduce a memory-efficient filter construction approach. We indicate our ALONE can be used as word representation sufficiently through an experiment on the reconstruction of pre-trained word embeddings. In addition, we also conduct experiments on NLP application tasks: machine translation and summarization. We combined ALONE with the current state-of-the-art encoder-decoder model, the Transformer, and achieved comparable scores on WMT 2014 English-to-German translation and DUC 2004 very short summarization with less parameters.
Introduction
Word embeddings are a fundamental component of modern NLP models, yet they present a significant memory bottleneck. In standard architectures, the embedding matrix scales linearly with the vocabulary size ($V$). For large-scale models, these embeddings can account for a substantial portion of the total parameters—often up to 25% in Transformer-based translation models. This linear scaling makes deployment on memory-constrained devices challenging.
This paper introduces ALONE (ALl word embeddings from ONE), a method that decouples the parameter count from the vocabulary size. By generating all word representations from a single shared vector using memory-efficient, non-trainable filters, ALONE drastically reduces the model's footprint without compromising performance on downstream tasks.
Methods
The ALONE architecture replaces the traditional large lookup table with a dynamic construction process.
The method consists of three primary components:
- Shared Base Embedding ($o$): A single, trainable vector shared by every word in the vocabulary.
- Word-Specific Filter Vectors ($m_w$): To differentiate words, the base vector is element-wise multiplied by a unique filter vector. These filters are not stored directly for each word. Instead, they are constructed from a small set of "source matrices" (codebooks). Each word is randomly mapped to a combination of vectors from these matrices, which are then summed to form $m_w$. These source matrices are randomly initialized and frozen during training.
- Shared Feed-Forward Network (FFN): The filtered embedding is passed through a 2-layer FFN with ReLU activation. This non-linear step is crucial for increasing the representational expressiveness, compensating for the shared nature of the base vector.
Memory-Efficient Filter Construction
To ensure the filters themselves do not become a memory bottleneck, ALONE uses a combinatorial approach. By randomly assigning combinations of frozen vectors from a small codebook, the model avoids collisions (leveraging the birthday paradox) while maintaining a minimal parameter count. The total number of trainable parameters in the embedding layer is thus determined by the FFN's dimensions rather than the vocabulary size.
Experiments and Results
The authors evaluated ALONE across three distinct settings: word embedding reconstruction, machine translation, and abstractive summarization.
Word Embedding Reconstruction
Using pre-trained 300-dimensional GloVe embeddings, ALONE demonstrated its ability to mimic high-dimensional representations. It achieved comparable Spearman's rank correlations on similarity benchmarks like SimLex-999 and WordSim-353 using only 1.4M parameters.
Machine Translation
On the WMT 2016 English-to-German task, a Transformer model utilizing ALONE achieved a BLEU score of 27.61, slightly outperforming the baseline Transformer (27.12 BLEU). This performance was achieved using only 8.4M embedding parameters, a 50% reduction from the baseline's 16.8M.
Abstractive Summarization
In summarization tasks using the DUC 2004 benchmark, ALONE achieved state-of-the-art ROUGE scores (e.g., 32.57 R-1). Most notably, it reduced the embedding parameter count from 8.3M to just 1.0M, an 8x reduction.
Discussion and Conclusion
A key insight from this work is that randomness is sufficient for word differentiation when paired with a trainable base embedding and a shared non-linear transformation. While the filters can be frozen, the base embedding $o$ must remain trainable to allow the model to adapt to the task.
One limitation noted is convergence speed; variants using binary masks for filters tend to converge slower than conventional models. However, ALONE represents a powerful, orthogonal approach to model compression that can be combined with pruning or quantization. It offers a promising path for scaling neural language models to even larger vocabularies without the proportional increase in memory costs.
Summary updated