Transfer Fine-Tuning: A BERT Case Study

Yuki Arase, Jun'ichi TsujiiEMNLP 2019 · DOI

TL;DR

This paper proposes a novel "transfer fine-tuning" approach to improve BERT's performance on semantic equivalence assessment tasks without increasing model size. By injecting phrasal paraphrase relations into BERT, the authors achieve superior results compared to larger BERT models, especially on tasks with limited training data, making powerful NLP models more accessible.

What Problem Does This Paper Address?

Natural Language Understanding (NLU) research heavily relies on tasks that assess semantic equivalence between sentences, such as paraphrase identification and semantic textual similarity (STS). While large pre-trained language models like BERT have revolutionized sentence representation learning, their effectiveness often comes with a significant increase in model size and computational cost. This poses a barrier to practical application, especially in scenarios with limited annotated data for fine-tuning. The authors aim to bridge this gap by developing a method that enhances BERT's ability to understand semantic equivalence without resorting to larger models, thereby improving efficiency and accessibility.

Methods

The core of this paper is the proposed "transfer fine-tuning" method. Instead of simply fine-tuning a pre-trained BERT model on a target task, the authors introduce a novel pre-training step that injects explicit semantic relations. Specifically, they leverage phrasal paraphrase alignments, derived from sentence pairs that are paraphrases of each other.

The method involves two main components:

  1. Phrasal Paraphrase Classification: This component focuses on learning relationships between aligned phrases within a sentence pair. After encoding the sentence pair with BERT, phrase embeddings are generated using max-pooling over the BERT hidden states corresponding to the phrase spans. These phrase embeddings are then used to classify whether the phrase pair represents a paraphrase.
  2. Sentential Paraphrase Classification: This component uses the standard BERT approach of feeding the [CLS] token's final hidden state into a classifier to determine if the entire sentence pair is a paraphrase.

The loss function is a combination of the losses from both phrasal and sentential paraphrase classification. This approach aims to provide BERT with a more direct and granular understanding of semantic equivalence, complementing its existing pre-training objectives.

Datasets

The authors utilize several datasets for their experiments:

  • Paraphrase Corpora: Datasets for training the transfer fine-tuning model were collected from various sources, including NIST OpenMT, Simple Wikipedia, Twitter URL corpus, and Para-NMT. These datasets provided both sentential and phrasal paraphrase pairs after alignment.
  • GLUE Benchmark: For evaluating the fine-tuned models, the authors used the General Language Understanding Evaluation (GLUE) benchmark. This benchmark includes:
  • Semantic Equivalence Assessment Tasks: MRPC (Microsoft Paraphrase Corpus), STS-B (Semantic Textual Similarity Benchmark), and QQP (Quora Question Pairs).
  • NLI Tasks: MNLI (Multi-Genre Natural Language Inference), RTE (Recognizing Textual Entailment), and QNLI (Question-Answering NLI).
  • Single-Sentence Tasks: SST (Stanford Sentiment Treebank) and CoLA (Corpus of Linguistic Acceptability).

Key Insights and Ideas

  • Injecting Phrasal Relations is Key: The paper strongly suggests that explicitly teaching BERT about phrasal paraphrases, in addition to sentential ones, significantly boosts its performance on semantic equivalence tasks. This goes beyond BERT's inherent ability to learn distributional sentence relationships.
  • Smaller Models Can Outperform Larger Ones: The proposed "transfer fine-tuning" method allows a smaller BERT model to achieve better or comparable results to a much larger BERT model on semantic equivalence tasks. This is a crucial finding for practical deployment.
  • Benefit for Low-Resource Scenarios: The method shows a disproportionately larger performance gain on tasks with limited fine-tuning data. This highlights its value for transfer learning, where obtaining large annotated datasets is often challenging.
  • Complementarity of Phrasal and Sentential Supervision: The ablation studies reveal that combining both phrasal and sentential paraphrase classification during fine-tuning yields the best results, indicating that these two forms of supervision are complementary.
  • Three-way Classification Advantage: Using a three-way classification (paraphrase, random, in-paraphrase) for phrasal paraphrases proved more effective than a simple binary classification, suggesting that learning to discriminate difficult negative examples is beneficial.

Key Results

  • The proposed "Transfer Fine-Tuning" model outperformed BERT-base on MRPC (+0.9 points) and STS-B (+2.7 points).
  • It even surpassed BERT-large (which has 3.1 times more parameters) by 0.6 points on MRPC and 1.4 points on STS-B.
  • On tasks with limited training data (e.g., 1k sentence pairs for MRPC and STS-B), the proposed model showed significant improvements over BERT-base, supporting hypothesis H2.
  • For NLI tasks, the model showed moderate improvements, supporting hypothesis H3, with a notable gain on RTE (+5.0 points over BERT-base), likely due to its limited training set size.
  • The model's performance degraded on single-sentence tasks (SST, CoLA), suggesting that its specialized training might reduce generality on tasks less related to sentence pairs.

Limitations and Future Work

  • Task Specificity: The authors acknowledge that their method's effectiveness is tied to the relevance between paraphrase discrimination and downstream tasks. The degradation on single-sentence tasks (SST, CoLA) suggests that the elaborate feature generation might sacrifice generality for specialized tasks.
  • Domain Mismatch: The performance on QQP was not improved, potentially due to a domain mismatch between the news-centric paraphrase corpora and the social QA forum data of QQP. Future work could involve generating multi-domain paraphrase corpora.
  • Model Size Investigation: The authors plan to investigate the effects of their method on different sizes of BERT models.
  • Alignment Quality: Improving the quality of phrase alignment models is another area for future research.
  • Simpler Feature Generation: An appendix explores a simpler feature generation method (mean-pooling) which also shows promising results, suggesting that even simpler approaches can be effective, and further investigation into the trade-offs between elaborate and simple feature generation is warranted.

Summary updated

BibTeX
@inproceedings{arase2019transfer,
  author = {Yuki Arase and Jun'ichi Tsujii},
  title = {{Transfer Fine-Tuning: A BERT Case Study}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1542},
}
Figure