Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-Tuning

Benjamin Minixhofer, Milan Gritta, Ignacio Iacobacci

TL;DR

Instead of relying on the standard Multi-Layer Perceptron (MLP) classification head when fine-tuning Transformer language models, this paper proposes replacing it with Gradient Boosted Decision Trees (GBDTs). By capturing representations generated during normal fine-tuning forward passes at zero additional neural compute cost, their proposed FreeGBDT consistently beats traditional MLP heads across Natural Language Inference (NLI) benchmarks, particularly on smaller datasets.

What Problem Does This Paper Address?

Fine-tuning a pretrained language model (such as RoBERTa) on a downstream classification task almost always follows the same recipe: attach a randomly initialized linear layer or small MLP to the final hidden state of the [CLS] token and train end-to-end.

However, in low-data regimes—where the number of training samples $n$ is smaller than or comparable to the hidden dimension $p$ ($n \le p$)—MLP heads are prone to overfitting and must learn to map representations from scratch. In contrast, Gradient Boosted Decision Trees (GBDTs) excel on dense numerical tabular features in sample-constrained settings. The authors ask: Can we replace the standard neural classification head with a GBDT, and can we do so without adding expensive extra forward passes?

Methods

The authors evaluate two integration strategies using RoBERTa-large:

[Tokens] ---> [ RoBERTa-large (Intermediate MNLI) ] ---> [CLS] Features
                                                              |
                  +-------------------------------------------+
                  |                                           |
         (During Fine-Tuning)                        (After Fine-Tuning)
                  |                                           |
            Log features                               One forward pass
           across epochs                                over train set
                  |                                           |
                  v                                           v
            [ FreeGBDT ]                               [ Standard GBDT ]
       (Trained on N x E samples)                    (Trained on N samples)
  1. Standard GBDT (Post-hoc Extraction): The Transformer is fine-tuned with a standard MLP head. Afterwards, an extra forward pass extracts the [CLS] embeddings across all $N$ training examples to train a LightGBM ensemble.
  2. FreeGBDT (Accumulated Training): Because neural network forward passes are computationally heavy, FreeGBDT caches the [CLS] representation produced during every forward pass across all $E$ training epochs. The tree ensemble is then trained on this accumulated pool of $N \times E$ samples. It costs zero extra neural network computation (hence "Free").

Unlike end-to-end tree architectures (e.g., neural decision trees), this approach requires no custom gradient formulations; the Transformer fine-tunes normally, and the GBDT simply takes over at evaluation time.

Datasets

The authors evaluate their approach on five Natural Language Inference benchmarks following intermediate supervised pretraining on the Multi-Genre NLI (MNLI) corpus:

  • CommitmentBank (CB): 250 training samples (3-class).
  • Recognising Textual Entailment (RTE): ~2.5k training samples (2-class).
  • Counterfactual NLI (CNLI): 6.6k training samples (3-class).
  • Adversarial NLI (ANLI): 162k training samples across rounds R1–R3 (3-class).
  • Question-answering NLI (QNLI): 104k training samples (2-class binary entailment).

No new datasets were introduced.

Key Insights and Ideas

  • Implicit Regularization via Representation Drift: A standard GBDT trained only on final-epoch features often overfits memorized training representations (e.g., dropping 1.5% below MLP performance on CNLI). FreeGBDT overcomes this because accumulating features across epochs exposes the trees to multiple "perturbed views" of each sample as weights update, acting as an effective data augmentation mechanism.
  • Capitalizing on Intermediate Pretraining: Intermediate task training (on MNLI) means the [CLS] token already maps to a semantically structured space before fine-tuning even starts ($t=0$). A tree ensemble can immediately partition these meaningful features, while an MLP must learn its parameters from a random initialization.
  • The Low-Data Advantage: Improvements are most pronounced on tiny datasets like CB ($n=250$) and RTE ($n\approx 2,500$), confirming theoretical expectations that tree-based algorithms shine when $n \le p$.

Key Results

  • Significant Aggregate Gains: On the development sets across 20 random seeds, FreeGBDT outperformed the standard MLP head on 4 out of 5 datasets and beat the standard GBDT on 5 out of 5 datasets ($p \approx 0.01$ via Wilcoxon signed-rank test).
  • Test Set Performance:
  • CB: FreeGBDT achieved 93.3% vs. 92.9% (MLP) and 91.3% (standard GBDT).
  • RTE: FreeGBDT achieved 87.8% vs. 87.5% (MLP).
  • CNLI: FreeGBDT scored 83.7% vs. 83.6% (MLP).
  • ANLI: FreeGBDT hit 57.6% vs. 57.4% (MLP). Crucially, on the challenging A2 round of ANLI, FreeGBDT reached 52.7%, establishing a new state-of-the-art over specialized methods like SMART (49.8%) and ALUM (52.1%).
  • QNLI: FreeGBDT tied the MLP at 94.3%. Because QNLI is a converted QA task that does not benefit from MNLI pretraining, the tree head did not provide an advantage, validating the prerequisite of aligned intermediate pretraining.
  • Zero Compute Overhead: The training time for LightGBM was negligible compared to Transformer backpropagation.

Limitations and Future Work

  • Modest Absolute Gains: Because top-layer Transformer representations are already rich and linearly separable, performance improvements are consistent but relatively modest (+0.2% to +0.6% on test sets).
  • Dependency on Intermediate Pretraining: FreeGBDT relies on the starting features ($t=0$) already containing relevant domain structure. On tasks where intermediate pretraining is unhelpful, it merely matches the baseline MLP.
  • Two-Stage Training: The tree ensemble cannot backpropagate gradients into the Transformer backbone during its own fitting phase.
  • Future Work: The authors suggest evaluating features drawn from multiple intermediate Transformer layers, extending the caching scheme to other backbones/modalities (e.g., ResNets in computer vision), and testing other classical classifiers like Support Vector Machines or Random Forests.

Summary updated

Abstract

Transfer learning has become the dominant paradigm for many natural language processing tasks. In addition to models being pretrained on large datasets, they can be further trained on intermediate (supervised) tasks that are similar to the target task. For small Natural Language Inference (NLI) datasets, language modelling is typically followed by pretraining on a large (labelled) NLI dataset before fine-tuning with each NLI subtask. In this work, we explore Gradient Boosted Decision Trees (GBDTs) as an alternative to the commonly used Multi-Layer Perceptron (MLP) classification head. GBDTs have desirable properties such as good performance on dense, numerical features and are effective where the ratio of the number of samples w.r.t the number of features is low. We then introduce FreeGBDT, a method of fitting a GBDT head on the features computed during fine-tuning to increase performance without additional computation by the neural network. We demonstrate the effectiveness of our method on several NLI datasets using a strong baseline model (RoBERTa-large with MNLI pretraining). The FreeGBDT shows a consistent improvement over the MLP classification head.
BibTeX
@inproceedings{minixhofer2021enhancing,
  author = {Benjamin Minixhofer and Milan Gritta and Ignacio Iacobacci},
  title = {{Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-Tuning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=lv1_Te3C5f},
}

Loading related content...

Figure