TokenDrop + BucketSampler: Towards Efficient Padding-free Fine-tuning of Language Models

Amrit Nagarajan, Anand RaghunathanEMNLP 2023 · DOI

Introduction

Fine-tuning large language models (LMs) is a cornerstone of modern NLP, yet it remains computationally expensive and prone to overfitting on small datasets. This paper introduces TokenDrop + BucketSampler, a framework designed to eliminate the "padding tax" and improve model generalization. By combining efficient sequence batching with a novel stopword-based regularization technique, the authors achieve up to 10.6x faster fine-tuning and significant accuracy improvements.

The Challenges of Fine-Tuning

The authors identify two primary bottlenecks in the fine-tuning pipeline:

  1. The Padding Tax: Standard random batching requires padding shorter sequences to match the longest one in a batch. These padding tokens consume GPU cycles without contributing to learning, often accounting for nearly 39% of processed tokens.
  2. The Overfitting Trap: Small downstream datasets often lead LMs to memorize specific inputs rather than learning generalizable features, making them sensitive to minor input variations.

Methodology

The proposed framework addresses these issues through two integrated components that attack both efficiency and accuracy simultaneously.

BucketSampler

Instead of random sampling, BucketSampler groups sequences of similar lengths into "buckets." To maximize efficiency, it employs Residual Batch Merging (RBM) to handle leftover samples without reintroducing excessive padding. To maintain stability on small datasets, the system uses BatchCap to gradually increase batch sizes and Learning Rate Modulation (LRM) to scale the learning rate dynamically.

TokenDrop

TokenDrop acts as both a regularizer and an efficiency booster. It randomly prunes "insignificant" tokens (stopwords like "the" or "is") during training. This forces the model to focus on core semantic meaning rather than specific function words, effectively creating "data diversity." Because sequences become shorter, padding requirements are further reduced.

Visualizing the reduction of padding tokens using BucketSampler and TokenDrop.

Experimental Results

The framework was evaluated using RoBERTa and ELECTRA models across the GLUE benchmark, SQuAD v1.1, and summarization tasks (XSum, CNN/DailyMail).

Efficiency and Accuracy Gains

The results demonstrate that padding can be reduced from approximately 39% to just 0.2%. This leads to a peak speedup of 10.61x on SST-2 and an average of 5.9x across the GLUE suite. Notably, these speedups do not come at the cost of performance; in many cases, accuracy improved by up to 1.17% for RoBERTa-Large.

Performance comparison and fine-tuning speedup across GLUE and SQuAD benchmarks.

Robustness and Inference

Models trained with TokenDrop exhibit increased resilience to noisy or grammatically incorrect inputs. Furthermore, the authors found that applying TokenDrop during inference can yield a 2.2x speedup with less than a 1% drop in accuracy, making it a viable strategy for real-time applications.

Conclusion and Limitations

While TokenDrop + BucketSampler offers substantial gains, it is not universally applicable. Tasks where stopwords are critical to the label (e.g., CoLA, which identifies grammatical correctness) may see degraded performance. Additionally, the benefits are most pronounced in fine-tuning rather than fixed-block pre-training. Future work may explore advanced optimizers to further improve hardware utilization during the initial training phases.

Summary updated

BibTeX
@inproceedings{nagarajan2023tokendrop,
  author = {Amrit Nagarajan and Anand Raghunathan},
  title = {{TokenDrop + BucketSampler: Towards Efficient Padding-free Fine-tuning of Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.782},
}
Figure