Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT

Jing Zhao, Yifan Wang, Junwei Bao, Youzheng Wu, Xiaodong He

Abstract

Transformer-based pre-trained models, such as BERT, have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, deploying these models can be prohibitively costly, as the standard self-attention mechanism of the Transformer suffers from quadratic computational cost in the input sequence length. To confront this, we propose FCA, a fine- and coarse-granularity hybrid self-attention that reduces the computation cost through progressively shortening the computational sequence length in self-attention. Specifically, FCA conducts an attention-based scoring strategy to determine the informativeness of tokens at each layer. Then, the informative tokens serve as the fine-granularity computing units in self-attention and the uninformative tokens are replaced with one or several clusters as the coarse-granularity computing units in self-attention. Experiments on GLUE and RACE datasets show that BERT with FCA achieves 2x reduction in FLOPs over original BERT with <1% loss in accuracy. We show that FCA offers a significantly better trade-off between accuracy and FLOPs compared to prior methods.

Introduction

Large-scale Transformer models like BERT have revolutionized NLP, but their "quadratic tax"—the fact that computational cost grows with the square of the sequence length—makes them expensive to deploy in real-world, resource-constrained environments. While techniques like pruning and distillation help, they often involve a trade-off: either discard information entirely or undergo a complex training process.

Existing solutions generally fall into two camps. Knowledge Distillation involves training a smaller "student" model to mimic a "teacher," which is effective but requires massive compute and complex pipelines. Pruning identifies and removes redundant tokens or attention heads, but suffers from information loss; once a token is pruned, its unique contribution to the context is gone, which can hurt performance, especially in early layers.

In this paper, we introduce FCA (Fine- and Coarse-granularity hybrid self-attention), a method that slashes BERT's computational footprint by progressively "squashing" less important tokens into coarse-grained units rather than simply throwing them away.

The FCA Architecture

The core bottleneck in BERT is the Self-Attention mechanism, where every token attends to every other token, leading to $O(n^2)$ complexity. The authors argue that maintaining full-length token representations is progressively redundant as you move deeper into the network. FCA seeks a "middle way": keep the important tokens at high resolution (fine-grained) and compress the rest (coarse-grained).

The FCA architecture introduces a Granularity Hybrid Layer after the Multi-head Self-Attention (MHA) block. The process follows three main steps:

  1. Informativeness Scoring: The model calculates an "importance score" for each token by summing its attention weights across all heads. If many other tokens are "looking" at a specific token, that token is deemed highly informative.
  2. The "Gather" Operation: Based on these scores, tokens are split into two groups:
  3. Informative (Fine-grained): The top-$k$ tokens (including the special [CLS] token) are kept exactly as they are.
  4. Uninformative (Coarse-grained): The remaining tokens are aggregated into one or more "coarse units" using pooling (either simple average pooling or weighted average pooling based on their scores).
  5. Progressive Shortening: The value of $k$ (the number of fine-grained tokens) is learnable and decreases in deeper layers. By the time the data reaches the final layers, the sequence length is significantly shorter, reducing the workload for both the MHA and the Feed-Forward Network (FFN) sub-layers.

Experimental Results

The authors evaluated FCA on two major benchmarks: GLUE (General Language Understanding Evaluation) and RACE (a large-scale reading comprehension dataset).

Key results include:

  • Efficiency: FCA-BERT achieves a 2x reduction in FLOPs compared to the original BERT-base.
  • Accuracy: This speedup comes with a negligible performance hit—less than 1% accuracy loss across GLUE and RACE tasks.
  • Superiority over Pruning: FCA consistently outperformed PoWER-BERT (a state-of-the-art pruning method) at similar FLOP counts, proving that keeping "coarse" versions of uninformative tokens is better than deleting them.
  • Generalization: The method was successfully extended beyond BERT-base to BERT-large and ELECTRA, showing it is architecture-agnostic.

Discussion and Insights

FCA's success stems from Information Retention. Unlike pruning methods that delete tokens, FCA's coarse units act as a "summary" of the background information, preventing the model from losing broader context. Furthermore, the authors observed Progressive Redundancy: in early layers, importance is spread out, while in deeper layers, it concentrates heavily on a few key tokens. This justifies aggressive compression in later stages.

Additionally, by shortening the sequence, FCA reduces the cost of the Position-wise Feed-Forward Networks, which consume a significant portion of BERT's total FLOPs. Finally, FCA is orthogonal to distillation; applying FCA to a distilled model like DistilBERT accelerates it even further.

Limitations and Future Work

Current FCA is optimized for sequence-level classification where only the [CLS] token's final representation is required. For token-level tasks like Named Entity Recognition (NER), maintaining the full length of the Query ($Q$) while compressing Key ($K$) and Value ($V$) is a potential path, though initial experiments were unsatisfactory. Future work will also focus on hardware-specific acceleration to translate theoretical FLOP reductions into real-world wall-clock speedups.

Summary updated

Abstract

Transformer-based pre-trained models, such as BERT, have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, deploying these models can be prohibitively costly, as the standard self-attention mechanism of the Transformer suffers from quadratic computational cost in the input sequence length. To confront this, we propose FCA, a fine- and coarse-granularity hybrid self-attention that reduces the computation cost through progressively shortening the computational sequence length in self-attention. Specifically, FCA conducts an attention-based scoring strategy to determine the informativeness of tokens at each layer. Then, the informative tokens serve as the fine-granularity computing units in self-attention and the uninformative tokens are replaced with one or several clusters as the coarse-granularity computing units in self-attention. Experiments on GLUE and RACE datasets show that BERT with FCA achieves 2x reduction in FLOPs over original BERT with <1% loss in accuracy. We show that FCA offers a significantly better trade-off between accuracy and FLOPs compared to prior methods.
BibTeX
@inproceedings{zhao2022fine,
  author = {Jing Zhao and Yifan Wang and Junwei Bao and Youzheng Wu and Xiaodong He},
  title = {{Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=AhLvFmyGfoo},
}
Figure