SoftAdaClip: A Smooth Clipping Strategy for Fair and Private Model Training

Dorsa Soleymani, Ali Dadsetan, Frank Rudzicz

Introduction

Differential Privacy (DP) has become the gold standard for protecting sensitive data during model training, typically implemented via Differentially Private Stochastic Gradient Descent (DP-SGD). To prevent any single individual's data from exerting undue influence, DP-SGD "clips" the norm of each sample's gradient to a fixed threshold. However, this mechanism introduces a significant "fairness tax." Underrepresented groups or "hard-to-learn" samples often produce gradients with larger norms. Standard "hard clipping" scales all these gradients down to the exact same value, effectively erasing the unique learning signals from minority subgroups. This leads to "gradient misalignment," where the model ignores the needs of the few to satisfy the many. SoftAdaClip aims to bridge this gap between privacy and equity.

The SoftAdaClip Method

SoftAdaClip introduces a new differentially private training strategy that replaces traditional "hard" gradient clipping with a smooth, tanh-based transformation. The method is built on two key innovations:

1. Smooth Scaling

Instead of the standard hard clipping function—which leaves small gradients alone and abruptly truncates large ones—SoftAdaClip uses a scaling factor based on the hyperbolic tangent function: $$\alpha_i = \tanh\left(\frac{C}{\|g_i\|_2 + \epsilon}\right)$$ This ensures that even for very large gradients, the resulting norm is strictly bounded by $C$, satisfying DP requirements, while preserving the relative differences between gradients. By maintaining this "gradient hierarchy," the model can learn more nuanced patterns from minority groups that would otherwise be lost.

Comparison of hard clipping vs. SoftAdaClip's smooth tanh-based scaling factor.

2. Adaptive Thresholding

The clipping threshold $C$ is not static; it is updated dynamically during training to match a target quantile of the gradient distribution. This ensures the "soft" transformation stays in a range where it is most effective. Without this adaptivity, the benefits of the smooth transformation can be lost, particularly in "low-gradient" scenarios where gradients are naturally small and require tighter thresholds to distinguish between high-norm samples.

Experimental Evaluation

The researchers evaluated SoftAdaClip on three diverse, real-world datasets where privacy and fairness are paramount:

  • GOSSIS-1-eICU: Structured healthcare data used for ICU mortality prediction.
  • MIMIC-III: Clinical text used for predicting patient length of stay.
  • Adult Income: A standard fairness benchmark for predicting individual income levels.

Results and Discussion

SoftAdaClip significantly reduces the "fairness tax" of private machine learning. On the eICU dataset, the method demonstrated a clear reduction in the loss gap for both Gender and Age subgroups compared to standard DP-SGD and previous adaptive methods.

Loss gap reduction on the eICU dataset across Gender and Age subgroups.

Key insights from the results include:

  • Massive Fairness Gains: SoftAdaClip reduced subgroup disparities by up to 87% compared to standard DP-SGD and up to 48% compared to state-of-the-art Adaptive-DPSGD.
  • Improved Utility: In most cases, the method lowered overall loss across train, validation, and test sets, suggesting better generalization.
  • Statistical Significance: Using Wilcoxon signed-rank tests, the authors confirmed that the reduction in loss gaps was statistically significant ($p < 0.001$), whereas the difference between standard DP-SGD and previous adaptive methods often was not.

Limitations and Future Work

While SoftAdaClip shows great promise, it currently focuses on tabular and text data. Testing on computer vision or multi-modal datasets is a necessary next step. Additionally, while the method is adaptive, it introduces a heightened sensitivity to the initial clipping threshold ($C_0$) in low-gradient settings, which may require task-specific tuning. Future research will explore other smooth functions beyond tanh and integration with different privacy-preserving architectures.

Conclusion

SoftAdaClip demonstrates that the "fairness tax" of differential privacy is not an inevitability. By preserving the relative importance of individual gradients through smooth scaling and adaptive thresholding, we can train models that are both private and equitable.

Summary updated

Abstract

Differential privacy (DP) provides strong protection for sensitive data, but often reduces model performance and fairness, especially for underrepresented groups. One major reason is gradient clipping in DP-SGD, which can disproportionately suppress learning signals for minority subpopulations. Although adaptive clipping can enhance utility, it still relies on uniform hard clipping, which may restrict fairness. To address this, we introduce SoftAdaClip, a differentially private training method that replaces hard clipping with a smooth, tanh-based transformation to preserve relative gradient magnitudes while bounding sensitivity. We evaluate SoftAdaClip on various datasets, including MIMIC-III (clinical text), GOSSIS-eICU (structured healthcare), and Adult Income (tabular data). Our results show that SoftAdaClip reduces subgroup disparities by up to 87% compared to DP-SGD and up to 48% compared to Adaptive-DPSGD, and these reductions in subgroup disparities are statistically significant. These findings underscore the importance of integrating smooth transformations with adaptive mechanisms to achieve fair and private model training.
BibTeX
@misc{soleymani2025softadaclip,
  author = {Dorsa Soleymani and Ali Dadsetan and Frank Rudzicz},
  title = {{SoftAdaClip: A Smooth Clipping Strategy for Fair and Private Model Training}},
  year = {2025},
  eprint = {2510.01447},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.01447},
}
Figure