ShareBERT: Embeddings Are Capable of Learning Hidden Layers

Jia-Cheng Hu, Roberto Cavicchioli, Giulia Berardinelli, Alessandro CapotondiAAAI 2024 · DOI

Introduction

Pre-trained Language Models (PLMs) like BERT and GPT-3 have revolutionized Natural Language Processing, but their massive scale—often exceeding 100 million parameters—presents a significant barrier to deployment on memory-constrained edge devices. While techniques such as Knowledge Distillation (KD), pruning, and quantization have been developed to reduce model size, the embedding layer often remains a bottleneck, accounting for up to 30% of the total parameter count in compressed models. ShareBERT addresses this challenge by proposing a radical architectural shift: reusing the parameters of the embedding matrix to construct the hidden layers of the Transformer.

Methods: Parameter-Sharing Mechanisms

The core innovation of ShareBERT lies in two primary techniques that enable "parameter-sharing" between input embeddings and hidden layers:

  1. Embeddings Parameter-Sharing (EPS): This method partitions the columns of the embedding matrix. Specific groups of columns are then used to populate the weight matrices for the Feed-Forward (FF) and Self-Attention (SA) layers. In this paradigm, the "weights" of the encoder are essentially different views of the word embeddings.
  2. Virtual Embeddings Parameter-Sharing (VEPS): To overcome the physical limits of the embedding size (e.g., when the required layer parameters exceed the embedding dimension), the authors apply a linear projection to create a "virtual" embedding matrix. This allows the model to scale its hidden size independently of the vocabulary's embedding dimension.
The ShareBERT architecture utilizes a virtual embedding matrix derived from the primary embedding matrix to provide weights for the transformer layers.

ShareBERT typically utilizes a single encoder layer that is shared across all depths, with weights derived directly from the embedding matrix via EPS or VEPS. This structure significantly reduces the unique parameter count while maintaining the depth required for complex language tasks.

Experimental Setup

The models were pre-trained using Masked Language Modeling (MLM) on English Wikipedia and BookCorpus. Evaluation was conducted across several dimensions:

  • GLUE Benchmark: Standard suite for Natural Language Understanding (NLU) tasks.
  • Word Similarity & Analogy: To ensure that sharing parameters does not degrade the quality of word representations.
  • Edge Probing: To analyze the linguistic information captured within the hidden layers.
  • Downstream Applications: Including Neural Machine Translation (NMT) and Image Captioning.

Key Results and Performance

ShareBERT achieves extreme compression without the necessity of computationally expensive Knowledge Distillation. ShareBERT Small, with only 5 million parameters, achieves a 21.9x compression rate while retaining 95.5% of BERT Base's performance. ShareBERT Large, at 27 million parameters, matches 100% of BERT Base's performance while being 4x smaller.

Comparison of ShareBERT variants against other compressed BERT models on the GLUE benchmark.

Beyond standard NLU, the approach proved successful in cross-domain applications. In Image Captioning tasks, the ShareBERT-based model achieved results nearly identical to the baseline while being 3.48x smaller.

Insights: The Expressivity of Embeddings

A key finding of the paper is that embeddings are sufficiently expressive to "learn" the logic required for hidden layers. Reusing these parameters forces a tighter coupling between word representations and the functions that process them. Surprisingly, this parameter-sharing actually improved performance on word similarity and analogy tasks. The authors suggest that because the embeddings are used to build the encoder, they gain access to higher-level structural and semantic information during training, leading to more robust representations.

Limitations and Future Work

While ShareBERT drastically reduces the memory footprint, it does not necessarily reduce computation time. To maintain performance with shared parameters, the authors often increased the "virtual" hidden size, making ShareBERT Base roughly 4.5x slower than BERT Base in terms of GFLOPS. Future research may explore learning approximating functions to better simulate model depth or combining ShareBERT with 8-bit quantization and pruning to push compression boundaries even further.

Summary updated

BibTeX
@inproceedings{hu2024sharebert,
  author = {Jia-Cheng Hu and Roberto Cavicchioli and Giulia Berardinelli and Alessandro Capotondi},
  title = {{ShareBERT: Embeddings Are Capable of Learning Hidden Layers}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29781},
}
Figure