Introduction
Fine-tuning large language models (LMs) is a cornerstone of modern NLP, yet it remains computationally expensive and prone to overfitting on small datasets. This paper introduces TokenDrop + BucketSampler, a framework designed to eliminate the "padding tax" and improve model generalization. By combining efficient sequence batching with a novel stopword-based regularization technique, the authors achieve up to 10.6x faster fine-tuning and significant accuracy improvements.
The Challenges of Fine-Tuning
The authors identify two primary bottlenecks in the fine-tuning pipeline:
- The Padding Tax: Standard random batching requires padding shorter sequences to match the longest one in a batch. These padding tokens consume GPU cycles without contributing to learning, often accounting for nearly 39% of processed tokens.
- The Overfitting Trap: Small downstream datasets often lead LMs to memorize specific inputs rather than learning generalizable features, making them sensitive to minor input variations.
Methodology
The proposed framework addresses these issues through two integrated components that attack both efficiency and accuracy simultaneously.
BucketSampler
Instead of random sampling, BucketSampler groups sequences of similar lengths into "buckets." To maximize efficiency, it employs Residual Batch Merging (RBM) to handle leftover samples without reintroducing excessive padding. To maintain stability on small datasets, the system uses BatchCap to gradually increase batch sizes and Learning Rate Modulation (LRM) to scale the learning rate dynamically.
TokenDrop
TokenDrop acts as both a regularizer and an efficiency booster. It randomly prunes "insignificant" tokens (stopwords like "the" or "is") during training. This forces the model to focus on core semantic meaning rather than specific function words, effectively creating "data diversity." Because sequences become shorter, padding requirements are further reduced.
Experimental Results
The framework was evaluated using RoBERTa and ELECTRA models across the GLUE benchmark, SQuAD v1.1, and summarization tasks (XSum, CNN/DailyMail).
Efficiency and Accuracy Gains
The results demonstrate that padding can be reduced from approximately 39% to just 0.2%. This leads to a peak speedup of 10.61x on SST-2 and an average of 5.9x across the GLUE suite. Notably, these speedups do not come at the cost of performance; in many cases, accuracy improved by up to 1.17% for RoBERTa-Large.
Robustness and Inference
Models trained with TokenDrop exhibit increased resilience to noisy or grammatically incorrect inputs. Furthermore, the authors found that applying TokenDrop during inference can yield a 2.2x speedup with less than a 1% drop in accuracy, making it a viable strategy for real-time applications.
Conclusion and Limitations
While TokenDrop + BucketSampler offers substantial gains, it is not universally applicable. Tasks where stopwords are critical to the label (e.g., CoLA, which identifies grammatical correctness) may see degraded performance. Additionally, the benefits are most pronounced in fine-tuning rather than fixed-block pre-training. Future work may explore advanced optimizers to further improve hardware utilization during the initial training phases.
Summary updated