UltraSparseBERT: 99% Conditionally Sparse Language Modelling

Peter Belcak, Roger Wattenhofer

Introduction

UltraSparseBERT is a novel BERT variant designed to achieve extreme conditional sparsity. By reorganizing feedforward networks (FFNs) into a tree-like structure, the model utilizes only 0.3% of its neurons (specifically 12 out of 4095) during inference for each layer. This architectural shift enables massive efficiency gains—up to a 78x speedup on CPUs and a 4.1x speedup on GPUs—while maintaining predictive performance comparable to standard dense BERT models.

The Challenge of Dense FFNs

In modern transformer-based architectures, Feedforward Networks house the vast majority of parameters. Standard models are "dense," meaning every neuron is computed for every input token, regardless of relevance. While Mixture of Experts (MoE) introduces sparsity by activating specific blocks of neurons, it remains a top-down approach.

UltraSparseBERT adopts a bottom-up strategy using Fast Feedforward Networks (FFFs). This allows for conditional execution at the individual neuron level, achieving significantly higher sparsity than typical MoE implementations.

Methodology: Fast Feedforward Networks

The core innovation lies in replacing standard FFNs with a balanced binary tree structure:

  • Conditional Execution: For any given input, the model performs a "Conditional Matrix Multiplication" (CMM). Starting at the root, a dot product determines whether the model traverses to the left or right child node.
  • Logarithmic Complexity: Because the model only follows a single path from root to leaf, an FFF with $n$ neurons only computes $O(\log_2 n)$ neurons per inference. For a layer with 4095 neurons, only 12 are "touched."
  • Architecture Refinement: Built on the "crammedBERT" framework, the design uses GeLU activations across all nodes, removes output biases, and allows multiple FFF trees to run in parallel (denoted as $K \times D$, where $K$ is the number of trees and $D$ is the depth).

Experimental Results

The models were evaluated on the GLUE (General Language Understanding Evaluation) benchmark, covering tasks from sentiment analysis (SST-2) to natural language inference (MNLI, QNLI, RTE).

Performance of UltraSparseBERT variants across the GLUE benchmark compared to dense baselines.

Efficiency and Performance Trade-offs

Across most GLUE tasks, UltraSparseBERT performed on par with BERT-base. While the CoLA task (linguistic acceptability) proved challenging for highly sparse models, the UltraSparseBERT-1x11 configuration retained approximately 98.6% of the predictive performance of the original BERT-base while ignoring 99.7% of feedforward parameters during the forward pass.

Hardware Acceleration

Standard deep learning frameworks are optimized for dense operations. To realize the theoretical benefits of sparsity, the authors developed custom kernels using Intel MKL for CPUs and OpenAI Triton for GPUs.

Inference speedup factors for UltraSparseBERT-1x11 compared to BERT-base-4095 on CPU and GPU.

As shown in the speedup analysis, the custom implementations provide substantial real-world gains, particularly on CPUs where the tree-traversal logic avoids the heavy overhead of dense matrix multiplication.

Limitations and Future Outlook

Despite the impressive speedups, a performance gap remains in specific tasks like CoLA, suggesting that certain linguistic reasoning may still benefit from denser computation. Furthermore, these gains currently rely on specialized low-level kernels. As software stacks like Triton and non-copying batched matrix multiplications mature, these sparse architectures are expected to become more accessible and even more efficient for production deployment.

Summary updated

Abstract

We present UltraSparseBERT, a BERT variant that uses 0.3% of its neurons during inference while performing on par with similar BERT models. UltraSparseBERT selectively engages just 12 out of 4095 neurons for each layer inference. This is achieved by reorganizing feedforward networks into fast feedforward networks (FFFs).To showcase but one benefit of high sparsity, we provide an Intel MKL implementation achieving 78x speedup over the optimized feedforward baseline on CPUs, and an OpenAI Triton implementation performing forward passes 4.1x faster than the corresponding native GPU implementation. The training and benchmarking code is enclosed.
BibTeX
@inproceedings{belcak2024ultrasparsebert,
  author = {Peter Belcak and Roger Wattenhofer},
  title = {{UltraSparseBERT: 99\% Conditionally Sparse Language Modelling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-short.10},
}

Loading related content...

Figure