Fairness penalties added to a neural network's training loss are usually optimized with Adam, like everything else. They are not like everything else. A group-fairness penalty depends on the parameters only through a handful of group-level averages, such as each group's positive-prediction rate or its precision. Its curvature, meaning how sharply the penalty bends as parameters move, is therefore almost entirely low-rank: at most a few directions in parameter space matter. We propose to exploit this structure directly. We would compute an exact, cheap diagonal curvature estimate for the fairness term by pulling back the curvature of the penalty from the small space of group statistics into parameter space. We would then use it as a separate per-parameter step-size correction next to the usual task-loss preconditioner. Because this needs only a few extra gradient evaluations and no random probing, it avoids the noise that makes generic diagonal-Hessian optimizers unreliable at small batch sizes.
The question matters because the known failures of differentiable fairness are optimization failures. Ratio-based criteria such as predictive parity stall under first-order training, fairness gradients are noisy when mini-batches contain few members of a group, and tuning the penalty weight takes an expensive search. A curvature correction built from the structure of the fairness objective could address all three at once. If it does not, a careful negative result would still tell practitioners that these failures are not a matter of conditioning.
Core Claim
Write the fairness penalty as $R(\theta) = r(s(\theta))$. Here $s(\theta) \in \mathbb{R}^K$ collects the $K$ group statistics the criterion depends on, for example one soft positive rate per group for demographic parity, or one soft precision per group for predictive parity. The function $r$ is the violation or projection penalty applied to them. The chain rule splits the parameter Hessian into two parts:
$$\nabla_\theta^2 R = J^\top \nabla_s^2 r \, J + \sum_{k=1}^{K} \frac{\partial r}{\partial s_k} \nabla_\theta^2 s_k .$$
In this expression, $J \in \mathbb{R}^{K \times N}$ is the Jacobian of the statistics with respect to the $N$ parameters, and $\nabla_s^2 r$ is the small $K \times K$ curvature of the penalty in statistic space.
The first term is a Gauss-Newton-style matrix, the part of the Hessian that comes from the curvature of the outer penalty alone. It is positive semidefinite whenever $r$ is convex in the statistics. That holds for squared violations and for the Kullback-Leibler projection penalties in fairret. The second term is weighted by $\partial r / \partial s_k$, which is exactly zero at a fair solution. So the dropped term vanishes where it matters most, in the same way the residual term vanishes in classical Gauss-Newton.
The diagonal of the first term can be computed exactly from $K$ statistic gradients:
$$\mathrm{diag}(J^\top M J) = \sum_{k,l} M_{kl}\, J_k \odot J_l ,$$
where $M = \nabla_s^2 r$, $J_k$ is the gradient of statistic $k$, and $\odot$ is elementwise multiplication. For binary groups and standard criteria, $K$ is between 2 and 4. The claim to test is that preconditioning the fairness gradient with this "statistic-space Gauss-Newton diagonal" gives three things over Adam and over generic diagonal-Hessian optimizers: faster and more reliable descent on ratio criteria, lower sensitivity to batch size, and a Pareto frontier that is less sensitive to the penalty weight.
Research Program
The optimizer. For a ratio statistic $s_a = n_a / d_a$, meaning a soft count of true positives over a soft count of predicted positives in group $a$, the Jacobian row is $(\nabla n_a - s_a \nabla d_a)/d_a$. It therefore comes from weighted-sum backward passes over the batch. The optimizer, which we call SGN, maintains two things. The first is a running estimate of the numerators and denominators across batches. The online state tracking in Differentiable Optimization Layers for Guaranteed Fairness in Deep Learning motivates this choice: $\partial r/\partial s$ and $M$ are evaluated on low-variance accumulated statistics, while the Jacobian $J$ comes from the current batch. The second is an exponential moving average of $\mathrm{diag}(J^\top M J)$. The task loss keeps its own diagonal preconditioner: either Adam's second moment or a sampled-label Fisher diagonal, which is the Gauss-Newton diagonal of cross-entropy. The final update divides the combined gradient by $D_{\text{task}} + \mu D_{\text{fair}} + \lambda$, where $\mu$ is the penalty weight and $\lambda$ is a damping constant.
Analysis. Three analyses accompany the optimizer.
First, we will characterize how $D_{\text{fair}}$ treats large-margin points, meaning samples far from the decision boundary. Understanding Fairness Surrogate Functions in Algorithmic Fairness identifies these points as the source of the gap between the smooth surrogate and the true thresholded metric, and handles them with a hand-tuned balancing factor. Sigmoid-type soft statistics give these points small Jacobian entries. We will check whether curvature-scaled steps shrink the surrogate-fairness gap without that factor.
Second, we will measure how much of the true fairness Hessian diagonal the Gauss-Newton part captures during training. We compare against a Hutchinson estimate on small networks. Hutchinson's estimator recovers a Hessian diagonal by averaging products with random sign vectors.
Third, we will derive the variance of the estimator as a function of per-group batch counts. We expect it to be lower than Hutchinson noise, since the only randomness is which samples land in the batch.
Scope and limits. The cost grows with $K$. For intersectional groups we will test $K$ up to about 16 (sex × race on ACSIncome) and report where the extra passes stop paying off. Relative to prior curvature work in fairness, the novelty is where the curvature lives. Fast & Fair uses Hessians with respect to inputs to speed up an adversarial inner loop for linear models, and never preconditions parameters. Newton Losses takes Newton steps in per-sample output space for sorting and shortest-path losses, and never touches group-aggregated statistics. Perceptual manifold curvature measures geometric bending of feature representations, not loss curvature. Generic diagonal-Hessian optimizers such as ADAHESSIAN and AGD have never been run on fairness objectives, and do not use the low-rank structure.
Baselines
Do-nothing and post-processing references. The simplest reference is an unconstrained model trained with AdamW. The second is the same model with group-specific thresholds chosen on validation data. Learning Non-Discriminatory Predictors proves that post-hoc correction can be badly suboptimal, but it remains the cheapest competitor and must be shown on the same Pareto plot. On Adult, Understanding Fairness Surrogate Functions reports a demographic parity violation of 0.182 for the unconstrained model.
Strongest first-order method. The strongest existing competitor is Adam on the fairret Kullback-Leibler projection penalty. fairret reports that this penalty consistently outperformed Adversarial Debias and Prejudice Remover, with almost no loss in the area under the ROC curve on linear criteria. No reported scalar result was found on Lacuna for it; the file reports only qualitative comparisons and plots. For bounded surrogates, the reference number is the General Sigmoid / Balanced Surrogate method from Understanding Fairness Surrogate Functions. It reports a demographic parity violation of 0.029 on Adult, with accuracy within 1–2% of the unconstrained model. The log-sigmoid surrogates from Scalable and Stable Surrogates are the stability reference on CelebA.
Generic curvature methods. These are AdaHessian and AGD applied to the combined loss. Reported overhead is about 3× AdamW runtime for AdaHessian and 7% for AGD on transformers, per AGD.
The exact-constraint method. Where it can be reimplemented on tabular data, we include the F-Layer from Differentiable Optimization Layers. It reports a 1.46% mean AUC gain over post-hoc projection on the SBA loan data.
Fair comparison protocol. Every method uses the same architecture, data splits, fairret statistic definitions, and batch sizes. Each gets the same number of hyperparameter trials over learning rate and penalty weight $\mu$, with SGN's damping $\lambda$ counted inside that budget. We compare at matched wall-clock time as well as matched steps, so the $K$ extra passes are charged against SGN.
What counts as beating the baselines. SGN wins if it achieves a strictly higher Pareto hypervolume in thresholded violation versus AUROC over 5 seeds on predictive parity. It must also match Adam+fairret within noise on demographic parity, where Adam is already reported to work well, at no more than 1.5× wall-clock.
Why not just…?
Why not just use Adam, which is already a diagonal preconditioner? Adam scales each parameter by the running magnitude of the total gradient, and the task gradient dominates it. The fairness direction therefore inherits a step size set by the task loss. That magnitude is not curvature, and the AGD results suggest the difference matters even on plain classification. The honest threat is that the gains come purely from splitting the fairness gradient off, not from curvature. We therefore include a "split Adam" ablation, with separate second-moment estimates for the task and fairness gradients. If split Adam matches SGN, we will report that separation, not curvature, is the ingredient.
Why not just take a Newton step in statistic space, as Newton Losses does for algorithmic losses? This is the most serious alternative. It captures the curvature of $M$ at almost no cost, and might be enough. It leaves the per-parameter scale of $J$ uncorrected, and $J$ is often concentrated in the last layer and in bias terms. We run it as an ablation: target statistics come from a damped Newton step on $r$, followed by first-order regression toward them. If it matches SGN, the paper's conclusion becomes that statistic-space curvature suffices, which is still a new result for fairness.
Why not just run AdaHessian or AGD on the combined loss? These methods need no new derivation. But Hutchinson probing of the whole Hessian is noisy exactly where fairness is noisy, at small per-group batch counts, and it spends its probes mostly on task curvature. The batch-size sweep below settles whether exploiting the structure is worth the specialization.
Why not just tune the penalty weight harder? Scalable and Stable Surrogates already reports that first-order fair training can require expensive search over this weight. We measure how sensitive each method's Pareto front is across a fixed grid of $\mu$ values. If Adam with a well-chosen $\mu$ matches SGN's best front, SGN's value is reduced to lower tuning cost, and we will say so.
Proposed Experiments
Tabular benchmarks. We use Adult, COMPAS, Bank Marketing, LawSchool, ACSIncome and CreditCard with multilayer perceptrons. The criteria are demographic parity, equal opportunity and predictive parity, implemented through the fairret library (Fairret library). Violations are measured after thresholding, alongside AUROC and accuracy. The vision benchmark is CelebA with a wide ResNet (smiling prediction, parity across sex), and FairFace if compute allows.
Decisive checks. There are four.
- Predictive parity. This is the criterion fairret flags as prone to stalling. The prediction is that SGN reaches a target violation in fewer steps and lower wall-clock time than Adam+fairret, and that split Adam and the statistic-space Newton ablation fall short.
- Batch-size sweep. We vary the batch size from 32 to 1024. The prediction is that SGN's thresholded violation degrades less than Adam's and AdaHessian's as batches shrink.
- Surrogate-fairness gap. This is the difference between the smooth surrogate and the thresholded metric. We track it over training and relate it to the fraction of the update carried by large-margin points, comparing against the Balanced Surrogate method's 0.029 on Adult.
- Intersectional scaling. On ACSIncome sex × race, we report the value of $K$ at which the extra passes erase the gains.
Failure criterion. A negative outcome on the first check, where the ablations match the full method, would falsify the core claim that parameter-space fairness curvature is the missing ingredient.
References
- fairret: a Framework for Differentiable Fairness Regularization Terms
- Differentiable Optimization Layers for Guaranteed Fairness in Deep Learning
- Understanding Fairness Surrogate Functions in Algorithmic Fairness
- Fast & Fair: Efficient Second-Order Robust Optimization for Fairness in Machine Learning
- Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms
- Predicting and Enhancing the Fairness of DNNs With the Curvature of Perceptual Manifolds
- ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning
- AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrix
- Learning Non-Discriminatory Predictors
- Scalable and Stable Surrogates for Flexible Classifiers with Fairness Constraints
- Fairret: a Flexible Pytorch Library for Fairness