Abstract
Transfer learning has become the dominant paradigm for many natural language processing tasks. In addition to models being pretrained on large datasets, they can be further trained on intermediate (supervised) tasks that are similar to the target task. For small Natural Language Inference (NLI) datasets, language modelling is typically followed by pretraining on a large (labelled) NLI dataset before fine-tuning with each NLI subtask. In this work, we explore Gradient Boosted Decision Trees (GBDTs) as an alternative to the commonly used Multi-Layer Perceptron (MLP) classification head. GBDTs have desirable properties such as good performance on dense, numerical features and are effective where the ratio of the number of samples w.r.t the number of features is low. We then introduce FreeGBDT, a method of fitting a GBDT head on the features computed during fine-tuning to increase performance without additional computation by the neural network. We demonstrate the effectiveness of our method on several NLI datasets using a strong baseline model (RoBERTa-large with MNLI pretraining). The FreeGBDT shows a consistent improvement over the MLP classification head.
Introduction
In modern NLP, the standard fine-tuning procedure involves appending a simple Multi-Layer Perceptron (MLP) head to a pretrained Transformer model. While effective, this approach may not be optimal for Natural Language Inference (NLI) tasks where training data is limited. In such low-data regimes, the high dimensionality of latent representations (like the [CLS] token) relative to the number of samples can lead to overfitting or suboptimal performance in MLPs.
Gradient Boosted Decision Trees (GBDTs) are renowned for their performance on dense numerical features in tabular data, particularly when the sample-to-feature ratio is low. This research investigates whether GBDTs can serve as a more robust classification head for Transformer-based NLI models.
Methodology: From MLP to FreeGBDT
The researchers propose two methods for integrating GBDTs into the Transformer pipeline. The first, Standard GBDT, involves extracting features from a fully fine-tuned Transformer and training a GBDT on these static representations.
The second, more innovative approach is FreeGBDT. This method harvests the [CLS] token features from every forward pass across all training epochs. By doing so, the GBDT is trained on a "trajectory" of features as the model evolves, effectively providing a form of data augmentation and regularization without any additional neural network computation.
Experimental Results
The effectiveness of FreeGBDT was evaluated using a RoBERTa-large base model pretrained on MNLI across five NLI benchmarks: ANLI, CNLI, RTE, CB, and QNLI.
FreeGBDT consistently outperformed the standard MLP head on four out of five datasets. The gains were most pronounced on smaller datasets like CB (CommitmentBank), which contains only 250 training samples, and on challenging adversarial sets like ANLI. On the CNLI (Counterfactual NLI) dataset, FreeGBDT showed a significant +1.5% improvement over the standard GBDT approach.
Interestingly, on the largest dataset (QNLI), the MLP head maintained a slight edge, suggesting that the advantages of GBDTs are most relevant when data is scarce relative to feature dimensionality.
Conclusion
Replacing the standard MLP head with a GBDT—specifically using the FreeGBDT method—offers a "free" performance boost for Transformer models on NLI tasks. By recycling the features generated during the fine-tuning process, FreeGBDT captures the model's learning trajectory, leading to better generalization and state-of-the-art results on difficult benchmarks like ANLI A2. This approach highlights the value of the internal representations produced during training as a rich source of data for ensemble-based classifiers.
Summary updated