TL;DR
This paper introduces EELBERT, a novel approach to compress transformer-based language models like BERT by replacing the traditional, memory-intensive input embedding layer with a dynamic, on-the-fly computation. This method significantly reduces model size with minimal impact on performance, making powerful NLP models more accessible for resource-constrained environments.
What Problem Does This Paper Address?
The field of Natural Language Processing (NLP) has seen tremendous progress driven by large, pre-trained transformer models such as BERT. While these models achieve state-of-the-art results, their sheer size poses a significant challenge for deployment, especially on devices with limited memory, storage, and computational power (e.g., smartphones, IoT devices). A substantial portion of these models' size comes from the input embedding layer, which maps discrete tokens to dense vector representations. Standard model compression techniques like pruning and quantization often struggle to effectively reduce the size of this specific layer, as it's typically a lookup table. This paper aims to tackle this bottleneck by proposing a method to eliminate the need for a static embedding lookup table altogether.
Methods
The core innovation of EELBERT lies in its "embeddingless" architecture. Instead of storing a large embedding lookup table, EELBERT dynamically computes token embeddings at runtime. This is achieved through an n-gram pooling hash function. Here's a breakdown of the process:
- N-gram Extraction: For a given input token, the system extracts all possible n-grams (sequences of n characters) up to a predefined maximum length (N).
- Rolling Hash: A rolling hash function is used to generate a signature for each n-gram.
- Projection and Hashing: These n-gram signatures are then projected using hash seeds. The hash seeds are partitioned, and an outer product is computed between the n-gram signatures and the partitioned hash seeds. This step involves several transformations to bound the matrix values.
- Averaging and Concatenation: The projected values for each n-gram are averaged to produce a d-dimensional vector (where 'd' is the desired embedding dimension). These vectors from all n-grams are then concatenated to form the final token embedding.
Crucially, unlike standard BERT embeddings, these dynamic embeddings are not updated during training. This means the embedding layer is effectively a fixed function. The authors also explore using these dynamic embeddings as an initializer for trainable embedding layers in smaller models, showing promising results.
Datasets
- Training: The models were pre-trained on the OpenWebText dataset.
- Evaluation: Performance was evaluated on the GLUE benchmark, a collection of nine natural language understanding tasks. The authors note that for smaller models, they exclude the COLA and STSB tasks due to observed instability.
Key Insights and Ideas
- Embedding Layer as a Bottleneck: The paper clearly demonstrates that the input embedding layer is a significant contributor to model size, especially in smaller BERT variants, and is a prime target for compression.
- Dynamic Embeddings for Size Reduction: Replacing the lookup table with a deterministic hash function offers a substantial reduction in model size without requiring the embeddings to be learned or updated.
- Complementary Compression: EELBERT's approach is complementary to other compression techniques (pruning, quantization), suggesting it can be combined for even greater efficiency.
- Smaller Models Benefit Most: The benefits of dynamic embeddings are most pronounced for smaller models where the embedding layer constitutes a larger fraction of the total parameters.
- Deeper is Better for Small Models: For models with limited resources, a deeper architecture with dynamic embeddings can outperform a shallower model with trainable embeddings.
- Hash Function Matters for Small Models: While a random hash function works reasonably well for larger models, the proposed n-gram pooling hash function shows a performance advantage for smaller models, indicating that capturing some morphological similarity is beneficial.
Key Results
- Significant Size Reduction: EELBERT-base achieved a ~21% reduction in parameter count and a 21% reduction in exported model size compared to BERT-base, with minimal GLUE score regression.
- UNO-EELBERT: The smallest model, UNO-EELBERT, achieved a GLUE score within 4% of standard BERT-tiny while being 15x smaller (1.2 MB).
- Smaller Models: EELBERT-mini was ~3x smaller than BERT-mini with only 0.7% absolute regression. EELBERT-tiny was ~9x smaller than BERT-tiny with 0.5% absolute regression.
- Latency Trade-off: While EELBERT offers significant memory savings, it introduces increased inference latency, particularly for smaller models where the embedding computation becomes a more significant portion of the total time. For EELBERT-tiny, latency was observed to be ~2.3x higher than BERT-tiny.
- Hash Initialization: Using the n-gram pooling hash function as an initializer for trainable embeddings showed a 0.5% absolute increase in GLUE score for BERT-base compared to random initialization.
Limitations and Future Work
- Inference Latency: The primary limitation acknowledged is the increased inference latency due to dynamic embedding computation. The authors suggest architectural optimizations like sparse masks and caching to mitigate this.
- Pre-training vs. Exported Size: The size advantage of EELBERT is realized in the exported model. During pre-training, the models have similar sizes because the output embedding layer is still required.
- Future Work: The authors suggest exploring architectural optimizations for latency reduction, such as sparse masks and parallel computation of embeddings. They also propose investigating the learned hash function's role in model quality further.
Summary updated