Span Fine-tuning for Pre-trained Language Models

Rongzhou Bao, Zhuosheng Zhang, Hai Zhao

Introduction

Pre-trained language models (PrLMs) like BERT typically operate on sub-word tokens. While highly effective, these models often fail to explicitly capture higher-level linguistic structures such as phrases or named entities, collectively referred to as "spans." Previous research, including SpanBERT and ERNIE, attempted to address this by incorporating span-level information during the pre-training phase. However, this approach is computationally expensive, inflexible, and occasionally inconsistent across different downstream tasks.

We present Span Fine-tuning (SF), a novel method that injects phrase-level information during the fine-tuning phase rather than pre-training. This "plug-and-play" module achieves performance gains comparable to span-aware pre-trained models while reducing training time from weeks to hours.

Methodology

The Span Fine-tuning framework consists of three primary components: sentence segmentation, a hierarchical CNN module, and representation fusion.

The Span Fine-tuning (SF) architecture, illustrating the flow from subword tokens to span-level and sentence-level representations.

Sentence Segmentation

To avoid the overhead of complex linguistic parsers, SF utilizes a pre-sampled dictionary of common n-grams (up to 5 words) extracted from the Wikitext-103 dataset. Input sentences are segmented into spans using a simple greedy matching strategy.

Hierarchical CNN Module

The segmented spans are processed through a two-level CNN architecture:

  1. Level 1 (Span-level): A CNN-Maxpooling layer aggregates sub-token embeddings from the PrLM into a single vector for each identified span.
  2. Level 2 (Sentence-level): A second CNN-Maxpooling layer aggregates these span vectors into a unified sentence-level representation.

Representation Fusion

The resulting span-enhanced sentence vector is concatenated with the standard [CLS] token representation from the PrLM. This combined representation is then passed to the final classification layer for the specific downstream task.

Evaluation and Results

The SF method was evaluated on the GLUE benchmark, SNLI, and CoNLL-2003 (NER).

Performance and Efficiency

Adding SF to BERT-Base resulted in an average GLUE score improvement of 1.1 points. When applied to RoBERTa-Large, the model achieved an average score of 89.8, with significant gains on tasks like RTE (a 4% improvement).

Crucially, SF is highly efficient. While pre-training a span-aware model like SpanBERT requires approximately 32 days on 32 V100 GPUs, SF achieves comparable results in just 12 hours on 2 Titan RTX GPUs.

Discussion and Insights

A key finding is that span information is highly effective when introduced during fine-tuning, allowing the model to learn task-specific representations. Furthermore, the size of the n-gram dictionary is a critical hyperparameter.

Accuracy on the CoLA task as a function of n-gram dictionary size, showing a performance peak at 20k entries.

As shown in the ablation studies, a dictionary that is too small fails to capture meaningful phrases, while an excessively large dictionary (over 500k entries) tends to over-segment or combine words into meaningless strings. The optimal "sweet spot" was identified between 20k and 200k entries.

Limitations

Current limitations include the reliance on a static dictionary, which may not generalize to specialized domains (e.g., medical or legal text), and the use of a simple greedy segmentation strategy. Future work could explore dynamic segmentation and alternative architectures for the span-aggregation layer.

Summary updated

BibTeX
@inproceedings{bao2021span,
  author = {Rongzhou Bao and Zhuosheng Zhang and Hai Zhao},
  title = {{Span Fine-tuning for Pre-trained Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.169},
}
Figure