SkipBERT: Efficient Inference with Shallow Layer Skipping

Jue Wang, Ke Chen, Gang Chen, Lidan Shou, Julian J. McAuley

Abstract

Large-scale pre-trained language models like BERT have revolutionized natural language processing but remain computationally expensive for real-time deployment. We present SkipBERT, a framework that accelerates inference by replacing the computationally intensive lower layers of BERT with a lightning-fast Precomputed Lookup Table (PLOT) of n-gram representations. By skipping shallow layers and incorporating early exit mechanisms, SkipBERT achieves up to a 65% reduction in latency on the GLUE benchmark while maintaining competitive accuracy.

Introduction

While models like BERT provide state-of-the-art performance, their multi-layer transformer architecture imposes significant latency. Existing compression techniques, such as knowledge distillation and early exiting, often struggle with the "cold start" problem: the necessity of executing initial layers to build basic syntactic and semantic context. SkipBERT is motivated by the observation that lower transformer layers primarily focus on local, short-distance relationships. By treating these shallow layers as a cacheable resource, SkipBERT avoids redundant computations for common n-grams.

Methodology

The SkipBERT framework introduces a "sandwich" approach to efficiency, optimizing both the bottom and top of the model architecture.

The Precomputed Lookup Table (PLOT)

The core innovation is the PLOT, which stores precomputed hidden states for millions of tri-grams (3-token sequences). During inference, the first $L$ transformer layers are bypassed. Instead, the system performs a high-speed lookup to retrieve representations for the n-grams present in the input text.

Chunk Aggregation and Global Context

Because tokens appear in multiple overlapping tri-grams, SkipBERT employs a sigmoid-based gating mechanism for chunk aggregation. This process merges different local views into a single, coherent representation for each token. These representations are then processed by a reduced number of "global" transformer layers designed to handle long-range dependencies.

SkipBERT architecture featuring the Precomputed Lookup Table (PLOT), chunk aggregation, and early exit mechanisms.

Training and Storage

SkipBERT is trained using a two-stage distillation process, first on a general corpus and then on task-specific data, to ensure the student model matches the teacher's performance. To manage the storage requirements of the PLOT (which can exceed 100GB), the system utilizes memory-mapped files (mmap). This allows the operating system to dynamically manage RAM and SSD usage, ensuring fast access to frequent n-grams without requiring massive physical memory.

Key Results

SkipBERT demonstrates superior efficiency across the GLUE and SQuAD benchmarks.

Latency and Accuracy

On GLUE tasks, SkipBERT achieves a 65% reduction in inference latency without sacrificing accuracy relative to the BERT-base teacher. Even in aggressive configurations using only two global layers, the model retains 95% of its original accuracy. On the SQuAD v1.1 reading comprehension task, SkipBERT significantly outperforms standard compressed baselines like DistilBERT and TinyBERT in both F1 and Exact Match scores.

Performance Curves

A critical advantage of SkipBERT is its "flatter" accuracy-latency curve. Because the initial representations retrieved from the PLOT are already partially contextualized, the model maintains higher accuracy even when forced to exit early.

Accuracy vs. Latency curves for SkipBERT and TinyBERT on MRPC and SST-2 tasks.

Discussion and Conclusion

SkipBERT proves that the "meaning" captured by shallow transformer layers is stable enough to be precomputed. By combining shallow layer skipping via PLOT with early-exit strategies, the model creates a highly efficient inference pipeline. While the storage overhead of the lookup table is a consideration for edge devices, the significant gains in speed and the robustness of the n-gram backoff strategy make SkipBERT a compelling solution for high-throughput NLP applications. Future work will explore PLOT compression and the use of discontinuous n-grams to further refine the skipped representations.

Summary updated

BibTeX
@inproceedings{wang2022skipbert,
  author = {Jue Wang and Ke Chen and Gang Chen and Lidan Shou and Julian J. McAuley},
  title = {{SkipBERT: Efficient Inference with Shallow Layer Skipping}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.503},
}
Figure