FairDP: Certified Fairness with Differential Privacy

Khang Tran, Ferdinando Fioretto, Issa Khalil, My T. Thai, Linh Thi Xuan Phan NhatHai Phan

TL;DR

FairDP is a novel training framework that simultaneously provides rigorous differential privacy (DP) guarantees and certified group fairness without severely degrading model accuracy. By training group-specific models independently and progressively aggregating their knowledge, it turns DP noise from a utility-destroying nuisance into a tool for statistically bounding and certifying fairness metrics.

What Problem Does This Paper Address?

As machine learning (ML) models are increasingly deployed in high-stakes domains like healthcare, finance, and criminal justice, ensuring they are both private and fair has become paramount. However, achieving both simultaneously is notoriously difficult.

Differential Privacy (DP) protects individual data privacy by adding random noise during training (typically via DP-SGD). Unfortunately, standard DP mechanisms often exacerbate group unfairness. Because noise affects gradient updates uniformly, it disproportionately impacts minority or underrepresented groups, widening the accuracy gap between groups and worsening model bias.

While previous works have attempted to balance privacy and fairness, they share a critical limitation: they cannot provide formal, certified guarantees for both properties at the same time while maintaining high model utility. FairDP bridges this gap by providing a unified framework that delivers both $(\epsilon, \delta)$-DP and a certified bound on group fairness ($\tau$-fairness).

Methods

FairDP relies on two core components: fairness-aware DP training and a DP Monte Carlo (DP-MC) approximation for inference-time certification.

  Dataset D ---> Split into K Groups (D_1 ... D_K) 
                     |
                     v
             Train K Group Models Independently (with DP Noise)
                     |
                     v
             Progressively Aggregate & Distill Knowledge
                     |
                     v
             Final DP-MC Sampling & Ensemble Inference
  1. Fairness-Aware DP Training: Instead of training on the entire dataset at once—where majority groups can drown out minority groups—FairDP splits the data into $K$ disjoint datasets based on protected attributes (e.g., race or gender). It trains independent models for each group. Crucially, it applies group-wise gradient clipping to bound each group's contribution, ensuring equal influence during the aggregation step.
  2. Knowledge Distillation: At each training step, the parameters of these group-specific models are averaged and propagated back to each group. This progressive integration acts as a form of knowledge distillation, helping the model generalize well across all groups and preserving overall utility.
  3. DP Monte Carlo & Ensemble Inference: At the final training step, FairDP uses a novel DP-MC sampling process. It injects DP-preserving noise into the scoring function's gradients and splits the final batch into $N$ micro-batches to generate an ensemble of $N$ models. This is done without incurring extra privacy costs. At inference time, predictions are made via an ensemble average.
Fig. 2. A schematic overview of FAIRDP.

By bounding the weights of the final scoring layer ($M$) and leveraging the exact Gaussian noise injected for privacy, the authors mathematically derive a closed-form formula to certify the upper and lower bounds of the model's fairness metrics (such as demographic parity or equal opportunity).

Datasets

The authors evaluated FairDP on three widely used benchmark datasets for fairness and privacy research:

  • Adult Dataset: 48,842 instances; task is to predict whether income exceeds \$50k/year; protected attribute is Gender.
  • Default of Credit Card Clients (Default-CCC): 30,000 instances; task is to predict credit default; protected attribute is Gender.
  • UTK-Face Dataset: 23,795 face images; task is binary classification based on features; protected attribute is Race.

Key Insights and Ideas

  • Noise as a Friend, Not a Foe: Traditionally, noise in DP is viewed as a necessary evil that degrades model performance. FairDP flips this perspective by leveraging the randomness of DP-preserving noise to smooth the decision boundary, making it possible to statistically quantify and guarantee fairness bounds.
  • Privacy Enhances Fairness Certification: A fascinating theoretical and empirical finding is that stronger privacy guarantees (larger noise scale $\sigma$) actually lead to tighter fairness certifications (smaller $\tau$). The added randomness reduces the model's capacity to rely on subtle, biased correlations in the training data.
  • Decoupling via Group-Wise Training: By forcing the model to learn from distinct groups independently before aggregating, FairDP prevents the disparate impact typically introduced when standard DP-SGD is applied to heavily imbalanced datasets.

Key Results

FairDP consistently outperformed state-of-the-art baselines (including DP-SGD, DPSGDF, and FairSmooth) across all datasets:

  • Massive Fairness Gains: FairDP improved group fairness metrics by more than 65% on average compared to standard baselines. For instance, on the Adult dataset under strict privacy ($\epsilon = 0.5$), demographic parity unfairness dropped from 0.079 (clean model) to just 0.014.
  • Minimal Utility Loss: This fairness optimization came with a marginal utility drop—less than 4% on average in accuracy and ROC-AUC across the benchmark datasets.
  • Robust Certification: The empirical fairness certifications ($\tau_{emp}$) held up robustly even when the authors artificially introduced data distribution shifts ($\gamma$) between the training and testing sets.

Limitations and Future Work

The authors note two primary limitations:

  1. Data Scarcity in Minority Groups: FairDP assumes that every protected group has sufficient data to train a reasonable group-specific model. In real-world scenarios with extreme data sparsity for certain minority groups, utility might suffer. The authors suggest layering private data augmentation techniques on top of FairDP to mitigate this.
  2. Centralized Setting: The framework currently requires a centralized server to have access to the protected attributes to perform the initial data split. Future work could explore extending this methodology to decentralized or federated learning environments where protected attributes must remain local to the user.

Summary updated

Abstract

This paper introduces FairDP, a novel training mechanism designed to provide group fairness certification for the trained model's decisions, along with a differential privacy (DP) guarantee to protect training data. The key idea of FairDP is to train models for distinct individual groups independently, add noise to each group's gradient for data privacy protection, and progressively integrate knowledge from group models to formulate a comprehensive model that balances privacy, utility, and fairness in downstream tasks. By doing so, FairDP ensures equal contribution from each group while gaining control over the amount of DP-preserving noise added to each group's contribution. To provide fairness certification, FairDP leverages the DP-preserving noise to statistically quantify and bound fairness metrics. An extensive theoretical and empirical analysis using benchmark datasets validates the efficacy of FairDP and improved trade-offs between model utility, privacy, and fairness compared with existing methods. Our empirical results indicate that FairDP can improve fairness metrics by more than 65% on average while attaining marginal utility drop (less than 4% on average) under a rigorous DP-preservation across benchmark datasets compared with existing baselines.
BibTeX
@inproceedings{tran2025fairdp,
  author = {Khang Tran and Ferdinando Fioretto and Issa Khalil and My T. Thai and Linh Thi Xuan Phan NhatHai Phan},
  title = {{FairDP: Certified Fairness with Differential Privacy}},
  booktitle = {SaTML},
  year = {2025},
  doi = {10.1109/satml64287.2025.00058},
}
Figure