SOFTADACLIP: A SMOOTH CLIPPING STRATEGY FOR FAIR AND PRIVATE MODEL TRAINING

Dorsa Soleymani, Ali Dadsetan, Frank RudziczSubmission to ICLR 2026 · Rejected

TL;DR

Differential privacy (DP) typically degrades model fairness because standard hard gradient clipping squashes the stronger learning signals originating from underrepresented groups. SoftAdaClip replaces rigid threshold clipping with an adaptive, smooth hyperbolic tangent ($\tanh$) transformation that bounds gradient sensitivity while preserving relative gradient magnitudes. Across clinical text, structured healthcare, and census tabular benchmarks, SoftAdaClip reduces demographic subgroup loss disparities by up to 87% compared to DP-SGD and 48% compared to Adaptive-DPSGD, with negligible impact on overall accuracy.

What Problem Does This Paper Address?

Differentially Private Stochastic Gradient Descent (DP-SGD) bounds the privacy risk of individual records by enforcing two steps: per-sample gradient clipping to a fixed threshold $C$, followed by calibrated Gaussian noise injection.

However, DP-SGD frequently degrades accuracy on minority and underrepresented subpopulations—a phenomenon termed disparate impact or inequitable clipping. Underrepresented groups often exhibit distinct or harder-to-learn feature patterns, resulting in larger gradient norms. Standard "hard" clipping truncates every gradient norm exceeding $C$ to exactly $C$, regardless of whether the original norm was 20 or 1,000. Meanwhile, majority samples with smaller gradients remain unaffected. This uniform truncation disproportionately suppresses minority updates, aligning optimization trajectories almost exclusively with majority groups. Existing adaptive clipping algorithms adjust $C$ over time, but still rely on hard cutoffs that discard relative gradient magnitudes above the threshold.

Methods

SoftAdaClip integrates two core ideas: smooth nonlinear gradient scaling and differentially private adaptive thresholding.

Standard Hard Clip:   g_bar = g * min(1, C / ||g||)
SoftAdaClip:          g_bar = g * tanh(C / (||g|| + ε))
  • Smooth Scaling via $\tanh$: Instead of using the piecewise $\min(1, C/\|g\|)$ function, SoftAdaClip computes a scaling factor $\alpha_i = \tanh\left(\frac{C}{\|g_i\|_2 + \epsilon}\right)$ and transforms the gradient to $\bar{g}_i = \alpha_i \cdot g_i$.
  • Formal Privacy Guarantees: Because $\tanh(x) < \min(x, 1)$ for all $x > 0$, the scaled gradient norm satisfies $\|\bar{g}_i\|_2 \le C$. The sensitivity remains strictly bounded by $C$, ensuring differential privacy guarantees identical to standard clipping without relaxing mathematical privacy parameters ($\epsilon, \delta$).
  • Preserving Relative Differences: If two gradients have norms $1.1$ and $1.2$ with $C = 1.0$, hard clipping maps both to norm $1.0$. SoftAdaClip assigns them scaling factors of $\approx 0.72$ and $0.68$, preserving their relative differences and internal subgroup variability.
  • Adaptive Threshold Updates: The threshold $C$ is updated dynamically at each iteration using a DP estimate of the fraction of samples exceeding the clipping bound relative to a target quantile $\gamma$.

Datasets

The authors evaluate SoftAdaClip across three distinct domains and modalities (no new benchmark datasets were created):

  1. MIMIC-III (Clinical Text): Length of Stay (LOS) prediction classified into four ordinal durations from ICU discharge summaries, evaluated across gender, age, and ethnicity using ClinicalBERT.
  2. GOSSIS-1-eICU (Structured Healthcare Tabular): In-hospital ICU mortality prediction from admissions across 204 hospitals, evaluated across gender and age.
  3. Adult Income (Tabular): Standard fairness benchmark predicting whether income exceeds \$50k/year, evaluated across gender and age on two architectures: a simple 2-layer MLP and a complex MLP with GroupNorm.

Key Insights and Ideas

  • Hard clipping creates an artificial information bottleneck: The primary driver of DP unfairness is not merely adding noise, but destroying relative gradient information through hard truncation. Smooth compression allows larger gradients to maintain their relative distinctions.
  • Smoothing requires adaptivity (and vice versa): In an ablation study evaluating Fixed Soft Clipping (smooth clipping with a static $C_0$), fairness improved in only 6 of 13 evaluated settings and worsened in 7. Soft clipping alone is insufficient; it must be paired with dynamic threshold estimation to ensure gradients stay in the sensitive scaling regime of the $\tanh$ curve.
  • Threshold scale matters for small-gradient tasks: When baseline gradient norms are naturally tiny (as seen in the simple Adult Income network), $\tanh$ slightly attenuates sub-threshold gradients. Tuning $C_0$ to smaller values ($0.01$ or $0.05$) restores the smooth clipping advantage and resolves the suppression.

Key Results

  • Subgroup Disparity Reductions: SoftAdaClip achieves the lowest subgroup loss gap in 7 of 9 evaluated dataset configurations. It reduces average subgroup disparity by 25% to 87% (average 56%) compared to DP-SGD, and by 8% to 48% (average 22%) compared to Adaptive-DPSGD.
  • Statistical Significance: Pairwise Wilcoxon signed-rank tests with Bonferroni correction confirm that SoftAdaClip’s fairness improvements over DP-SGD ($p < 0.0004$) and Adaptive Clipping ($p < 0.001$) are statistically significant, whereas standard DP-SGD and Adaptive Clipping show no significant difference from each other ($p = 1.0$).
  • Utility Preservation: Reductions in disparity come without severe utility penalties. Overall model accuracy remains within 1–2 percentage points of non-private models and drops by less than 1% compared to Adaptive-DPSGD across all datasets.

Limitations and Future Work

  • Modality Scope: The evaluation covers healthcare text, healthcare records, and tabular census data, but excludes computer vision and multimodal models.
  • Base Adaptive Strategy: SoftAdaClip builds exclusively upon the adaptive framework of Andrew et al. (2021). The effects of $\tanh$ smoothing on other adaptive or per-sample clipping mechanisms remain unexplored.
  • Hyperparameter Sensitivity in Low-Gradient Regimes: When baseline gradients are exceptionally small, SoftAdaClip requires tuning $C_0$ downward to avoid under-scaling signals, which adds optimization overhead.
  • Minor Accuracy Penalty: In isolated configurations, SoftAdaClip exhibits a slight accuracy reduction ($<1\%$) compared to Adaptive-DPSGD, reflecting a subtle utility-fairness trade-off.

Summary updated

Abstract

Differential privacy (DP) provides strong protection for sensitive data, but often reduces model performance and fairness, especially for underrepresented groups. One major reason is gradient clipping in DP-SGD, which can disproportionately suppress learning signals for minority subpopulations. Although adaptive clipping can enhance utility, it still relies on uniform hard clipping, which may restrict fairness. To address this, we introduce SoftAdaClip, a differentially private training method that replaces hard clipping with a smooth, tanh-based transformation to preserve relative gradient magnitudes while bounding sensitivity. We evaluate SoftAdaClip on various datasets, including MIMIC-III (clinical text), GOSSIS-eICU (structured healthcare), and Adult Income (tabular data). Our results show that SoftAdaClip reduces subgroup disparities by up to 87% compared to DP-SGD and up to 48% compared to Adaptive-DPSGD, and these reductions in subgroup disparities are statistically significant. These findings underscore the importance of integrating smooth transformations with adaptive mechanisms to achieve fair and private model training.
BibTeX
@misc{soleymani2025softadaclip,
  author = {Dorsa Soleymani and Ali Dadsetan and Frank Rudzicz},
  title = {{SOFTADACLIP: A SMOOTH CLIPPING STRATEGY FOR FAIR AND PRIVATE MODEL TRAINING}},
  year = {2025},
  eprint = {2510.01447},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.01447},
}
Figure