Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT

Jing Zhao, Yifan Wang, Junwei Bao, Youzheng Wu, Xiaodong He

Abstract: Jing Zhao, Yifan Wang, Junwei Bao, Youzheng Wu, Xiaodong He. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

2022.

Introduction

Transformer-based models like BERT have become the standard for Natural Language Processing (NLP). However, their self-attention mechanism carries a "quadratic efficiency tax": the computational cost grows at $O(n^2)$ relative to the input sequence length. Doubling a sentence length quadruples the required work, making these models expensive to deploy in real-time environments.

Existing solutions typically involve Knowledge Distillation (mimicking a larger model) or Pruning (deleting redundant tokens). While effective, pruning often leads to permanent information loss, particularly in early layers where global context is still being established. Researchers from JD AI Research have introduced FCA (Fine- and Coarse-granularity hybrid self-attention) to address this by compressing rather than deleting information.

Methodology: The FCA Framework

The core innovation of FCA is the treatment of tokens based on their "informativeness." Instead of a binary keep-or-delete approach, the model maintains a hybrid representation:

  1. Attention-Based Scoring: The model calculates an informativeness score for every token using self-attention maps. Tokens that are frequently "attended to" by others are deemed highly informative.
  2. The FCA Layer: Tokens are ranked by these scores. The top-$k$ tokens are preserved as fine-grained units. The remaining tokens are aggregated (via weighted pooling) into a small set of coarse-grained units.
  3. Progressive Shortening: As the model processes deeper layers, it progressively reduces the number of fine-grained tokens, reflecting the increasing redundancy of representations for classification tasks.
The FCA architecture across layers, illustrating how tokens are gathered into fine and coarse units to reduce sequence length while preserving global information.

Insights: Progressive Redundancy

A key observation driving FCA is that informativeness is not distributed uniformly across a model's depth. In early layers, attention is spread across many tokens as the model builds initial context. In later layers, attention concentrates heavily on a few key tokens (such as the [CLS] token).

Attention score distributions across layers, showing the increasing concentration of informativeness and variance in deeper layers, which justifies more aggressive compression.

This empirical finding justifies the "progressive shortening" strategy. By clustering uninformative tokens into coarse units rather than deleting them, FCA maintains a "global view" of the text, preventing the accuracy drops typically seen in aggressive pruning methods like PoWER-BERT.

Key Results

FCA-BERT was evaluated on the GLUE and RACE benchmarks, demonstrating superior efficiency-accuracy trade-offs:

  • Efficiency: FCA achieves a 2x reduction in FLOPs (Floating Point Operations) compared to the original BERT-base.
  • Accuracy: This 50% reduction in compute resulted in less than 1% loss in accuracy on the GLUE benchmark.
  • Compatibility: FCA is orthogonal to distillation; it can be applied to already-distilled models like DistilBERT to achieve further speedups.
  • Pooling Strategy: Weighted average pooling, where the informativeness score determines the weight in the cluster, consistently outperformed simple average pooling.

Limitations and Future Work

Currently, FCA is optimized for classification tasks where the final output is condensed into a single vector. It cannot be directly applied to sequence-labeling tasks (like Named Entity Recognition) or extractive Question Answering without modification, as these require full-length outputs for every token. Future work aims to adapt the hybrid mechanism for these generative and token-level tasks, as well as optimizing low-level CUDA kernels for better hardware acceleration on GPUs.

Summary updated

BibTeX
@inproceedings{zhao2022fine,
  author = {Jing Zhao and Yifan Wang and Junwei Bao and Youzheng Wu and Xiaodong He},
  title = {{Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=AhLvFmyGfoo},
}
Figure