Algorithmic Fairness for Gradient Boosted Trees

15 concepts8 papers

In fair machine learning, a fundamental tension exists between predictive power and architectural compatibility. For tabular data—the backbone of high-stakes decision-making in finance, healthcare, and governance—Gradient Boosted Decision Trees (GBDTs) are the undisputed industry standard. However, as noted by Shen et al., the vast majority of algorithmic fairness frameworks are designed exclusively for differentiable neural networks.

This incompatibility stems from a mathematical barrier: standard fairness interventions rely on backpropagating a fairness penalty through a model to update its parameters. Because decision trees are discrete step functions, their gradients are zero almost everywhere and undefined at split boundaries. This non-smooth nature makes it notoriously difficult to enforce fairness constraints, whether group-level or individual, using standard gradient-based optimization (Vargo et al.).

To bridge this gap, a new class of algorithms bypasses the non-differentiability of tree structures by shifting the focus of optimization from the internal parameters of the model to its continuous outputs.

The Core Insight: Exploiting the Boosting Objective

To understand why this approach works, we must look at how gradient boosting algorithms construct an ensemble. Unlike neural networks, which update weights via gradient descent, a GBDT builds an ensemble sequentially:

$$F_t(x) = F_{t-1}(x) + \gamma h_t(x)$$

where each new base tree $h_t(x)$ is trained to fit the pseudo-residuals of the loss function. Crucially, these pseudo-residuals are the derivatives of the loss function taken with respect to the current ensemble's predictions, not the tree parameters:

$$r_{it} = -\left[ \frac{\partial \mathcal{L}(y_i, F(x_i))}{\partial F(x_i)} \right]_{F(x) = F_{t-1}(x)}$$

Because the ensemble's raw output $F(x)$ is a continuous scalar value, we can compute the derivative of a fairness penalty with respect to $F(x)$. Once this derivative is computed, it can be added directly to the task's pseudo-residuals. The subsequent tree-building step then uses standard, non-differentiable splitting criteria (like variance reduction) to fit these modified targets. This elegant decoupling allows practitioners to inject fairness constraints into non-differentiable models without needing end-to-end differentiability.

Mechanism 1: Neural Adversaries as Gradient Providers

One prominent manifestation of this idea is Fair Adversarial Gradient Tree Boosting (FAGTB), introduced by Grari et al. and expanded by Detyniecki et al..

[ GBDT Predictor F(x) ] ---> Prediction F(x) ---> [ Differentiable Neural Adversary A ]
         ^                                                          |
         |                                                          v
   Fits Residuals <--- Combined Gradient <--- Compute Gradient w.r.t F(x)

The framework sets up a zero-sum game between two architectures:

  1. The Predictor: A non-differentiable GBDT that maps features $X$ to a prediction $F(x)$ to minimize label error.
  2. The Adversary: A fully differentiable neural network that attempts to predict a sensitive attribute $S$ (such as race or gender) using only the predictor's continuous output $F(x)$.

Because the adversary is a neural network, its loss function $\mathcal{L}_{adv}$ is fully differentiable with respect to its inputs. The algorithm computes the gradient of the adversary's failure to predict the sensitive attribute and adds it to the standard task loss gradient. When the GBDT constructs the next tree, it naturally steers its predictions away from patterns that reveal the protected attribute, satisfying strict constraints like Demographic Parity or Equalized Odds.

Mechanism 2: Smooth Surrogates and Constrained Optimization

An alternative to training an explicit adversary is to formulate fairness directly as a constrained optimization problem within the boosting loop. However, standard group fairness metrics (like differences in selection rates) rely on indicator functions, which are themselves non-differentiable.

To resolve this, Cruz et al. introduced FairGBM. Instead of using a neural network to approximate the fairness penalty, FairGBM replaces non-differentiable fairness metrics with smooth mathematical surrogates, such as cross-entropy. By applying a proxy-Lagrangian formulation, the algorithm dynamically adjusts the importance weights of fairness violations during the boosting process. This allows the model to compute exact fairness gradients with respect to the model outputs at each boosting iteration, achieving state-of-the-art fairness-performance trade-offs at a fraction of the computational cost of adversarial training.

Expanding the Paradigm

The strategy of adapting fairness constraints to non-differentiable models has expanded across multiple dimensions of algorithmic fairness:

  • Multivariate and Continuous Attributes: While early methods focused on binary sensitive attributes, frameworks like FairCOCCO developed by Liu et al. leverage kernel-based cross-covariance operators. This provides a differentiable regularizer capable of handling multiple, continuous, or multi-type sensitive attributes simultaneously within tree-based models.
  • Individual and Counterfactual Fairness: Moving beyond group-level metrics, Vargo et al. designed methods to enforce individual fairness, smoothing out the sharp, discrete jumps typical of tree models to ensure similar individuals receive similar predictions. For scenarios where retraining is impossible, Hatano et al. proposed a post-processing approach using cooperative game theory (Harsanyi dividends) to surgically remove unfair causal influences from pre-trained random forests.
  • High-Stakes Deployments: The practical necessity of these hybrid approaches is highly visible in clinical decision support. For instance, Tang et al. demonstrated that combining tree-based models with neural debiasing steps allows clinical models to maintain high predictive accuracy for liver transplant outcomes while significantly mitigating systemic demographic disparities.

By shifting the mathematical interface from the model's internal parameters to its continuous predictions, these techniques prove that end-to-end differentiability is not a prerequisite for rigorous bias mitigation. Practitioners can deploy the high performance of gradient boosted trees on tabular data without sacrificing equity.

Key Foundational Work

Editorial of Special Issue of ICDM 2019 (2020) — contributed 4 concept elements to this direction.
Achieving Fairness with Decision Trees: An Adversarial Approach (2020) — contributed 3 concept elements to this direction.
Fair Adversarial Gradient Tree Boosting (2019) — contributed 3 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 8 papers
  2. Version 1 · 2026-10-04 · 8 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_adversarial_gradient_boosting_60432318,
  title = {adversarial gradient boosting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adversarial-gradient-boosting/rdr_60432318714a4e61b519758629497b12},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_adversarial_gradient_boosting_60432318_v2,
  title = {adversarial gradient boosting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adversarial-gradient-boosting/rdr_60432318714a4e61b519758629497b12/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake