Most fair-learning methods commit to one definition of fairness when the model is trained. Two common definitions are demographic parity, where predictions have the same distribution in every group, and equalized odds, where predictions have the same distribution in every group among people who share the same true outcome. Changing the definition means retraining, and methods that avoid retraining usually need the sensitive attribute at prediction time. We propose a way around both constraints for sparse Gaussian processes. A Gaussian process (GP) is a Bayesian model that places a probability distribution over functions. Its sparse variant summarizes the whole dataset through a few hundred "inducing points": learned virtual inputs whose function values carry everything the model knows. In such a model the predicted mean at any input is a fixed linear function of the posterior mean over those inducing values.
That linearity has a direct consequence. For any fairness criterion stated as equal average scores within cells defined by group and label, the criterion becomes a linear constraint on an $M$-dimensional vector, where $M$ is the number of inducing points. Demographic parity, equal opportunity, equalized odds and predictive equality are all of this form. Enforcing any of them after training becomes a closed-form projection of the variational posterior, the Gaussian approximation to the posterior over inducing values that sparse GP training produces. The projection costs one small matrix solve. It needs the sensitive attribute only in a one-pass summary of the training data, never at test time. The fairness–accuracy trade-off for each metric is a straight line in parameter space, so it can be traversed in milliseconds.
Core Claim
Train one unconstrained sparse variational GP. Then, for each fairness metric a deployer might want, move its posterior as little as possible, measured by Kullback–Leibler (KL) divergence, until that metric's moment constraint holds. We claim this gives fairness–accuracy trade-offs that are close to per-metric retrained models, across both marginal and label-conditional metrics, from a single fit.
The mechanism is this. Let $q(u)=\mathcal{N}(m,S)$ be the variational posterior over inducing values, with inducing inputs $Z$. Let $a(x)=K_{ZZ}^{-1}k_Z(x)$ be the $M$-vector of basis weights, so the predictive mean is $\mu(x)=a(x)^\top m$. For each cell $(g,y)$ of sensitive group $g$ and label $y$, compute the average basis vector $\bar a_{g,y}$ in one $O(NM)$ pass over the data. Pooling over labels gives $\bar a_g$. Each metric is a small constraint matrix $C$ with target $b$. Demographic parity uses the single row $(\bar a_1-\bar a_0)^\top$ with $b=0$. Equalized odds uses one row $(\bar a_{1,y}-\bar a_{0,y})^\top$ for each $y$. The information projection that minimizes $\mathrm{KL}(q'\,\|\,q)$ subject to $Cm'=b$ keeps $S'=S$ and sets
$$ m'(t)=m-t\,S C^\top (C S C^\top)^{-1}(Cm-b),\qquad t\in[0,1], $$
where $t$ controls how much of the constraint violation is removed. This form follows from the standard result that KL-projecting a Gaussian onto a linear constraint on its mean shifts the mean in the metric of its own covariance.
The posterior covariance $S$ is the key ingredient. It moves the function mainly along directions the data leave uncertain, so accuracy should suffer less than under a naive Euclidean shift. It also explains why the conditional metrics fit here even though a prior cannot express them. Kernel dependence regularizers and Gaussian processes with applications to algorithmic fairness enforces demographic parity by modifying the GP prior. A prior cannot condition on labels it has not yet seen, so equalized odds is out of reach there. Our constraint instead acts on the posterior, after the labels are known.
Relation to Prior Work
Three works motivate the proposal. The Fair GP of Li et al. adds a Hilbert–Schmidt Independence Criterion penalty, a kernel measure of statistical dependence, to an exact GP. It handles demographic parity only, pays $O(N^3)$ cost, and names sparse approximations as open work. We differ in placing fairness in a post-training projection rather than the prior, which lets one model serve conditional metrics too. Fair Classification with Efficient and Post-hoc Controllable Fairness-Accuracy Trade-off shows that post-hoc control is valuable: its bi-level neural training gives millisecond threshold adjustment for demographic parity. We aim for the same controllability without bi-level training and across several metrics. Fairness Beyond Disparate Treatment & Disparate Impact constrains the covariance between the sensitive attribute and signed boundary distance for linear classifiers, and needs a new convex program for each constraint level. Our cell-mean constraints are a kernelized relative of that proxy, but they are solved in closed form after training.
The method itself comes from Gaussian Processes for Big Data and Scalable Variational Gaussian Process Classification. Both give the inducing-point posterior $q(u)$, minibatch training at $O(M^3)$ per step, and, for classification, a latent function with a Bernoulli likelihood.
Two works constrain the design. On Coresets for Fair Regression and Individually Fair Clustering proves that data summaries chosen without regard to fairness drop small groups, which breaks fairness feasibility. This warns us that ELBO-optimized inducing locations may leave minority cells poorly represented. A poorly represented cell makes $\bar a_{g,y}$ noisy and makes $CSC^\top$ badly conditioned. Fair regression with Wasserstein barycenters gives the optimal post-processing for exact distributional demographic parity. That optimum requires the group at prediction time, and it splits into separate problems for each label under conditional metrics. It is our main post-processing competitor. It is also a reminder that matching first moments is weaker than matching whole distributions.
Research Program
The first component is the projection machinery for regression and classification. For regression the constraint acts on the predictive mean directly. For classification we constrain group-and-label means of the latent score. This parallels Zafar's constraint on signed distances, but makes the true error-rate gaps a measured outcome rather than the objective. We then test a refinement that constrains mean predicted probabilities instead of latent scores. This refinement is nonlinear in $m$, so we handle it by iterating the linearized projection a few times.
The second component is a relaxed version with tolerance $|Cm'-b|\le\epsilon$ and a Pareto sweep over $t$. Because $m'(t)$ is linear in $t$, the entire accuracy–fairness curve for a metric costs one solve.
The third component analyzes when projection loses to retraining. Retraining lets the data re-fit the variance $S$ and the inducing locations $Z$; projection holds both fixed. We expect the gap to grow with how far $Cm$ starts from $b$, measured in the metric $(CSC^\top)^{-1}$. We will report that quantity alongside every result.
The fourth component studies inducing coverage of minority groups. We compare locations $Z$ from ELBO optimization with $Z$ initialized by group-and-label stratified sampling, borrowing the stratification idea from the fair coresets work. We then measure how coverage affects the conditioning of the projection and the accuracy paid for fairness. As an extension we sketch second-moment constraints, which are quadratic in $(m,S)$, to move from matching means toward matching distributions.
Baselines
The naive reference point is a standard sparse variational GP trained with the sensitive attribute removed, following Hensman et al. It establishes the unconstrained accuracy and the fairness gap that proxy features induce. It is run with the same inducing budget, kernel and optimizer as our base model; in fact it is our base model before projection. Every fairness method must show a large gap reduction relative to it.
The strongest in-processing competitor is a sparse variational GP retrained separately for each metric, with a minibatch penalty on that metric's squared cell-mean gap added to the ELBO. It shares the representation, data and compute of our method, and retraining lets it re-fit $S$ and $Z$, so it represents the accuracy ceiling for this model class. We give it a tuning budget of ten penalty weights per metric. We count the proposal as beating it if our Pareto curve lies within its confidence band while costing one fit in total instead of one fit per metric and per trade-off point.
On CRIME we also run the exact Fair GP of Li et al. for demographic parity. That paper reports an unfairness drop from 0.050 to 0.0024, with RMSE rising from 0.627 to 0.766. We must reach comparable unfairness with RMSE at or below 0.766.
The strongest post-processing competitor is the Wasserstein barycenter method of Chzhen et al. applied to the same base GP. For classification we add group-specific thresholding in the style of FairBayes, as used in Sakata and Fukuchi. The barycenter paper reports Kolmogorov–Smirnov fairness improvements of 50–80% over competing fair methods. With random-forest base models it reports a 53% average fairness gain for about a 12% increase in mean squared error. These methods use the sensitive attribute at test time, which ours does not. We therefore report them as a reference that holds an extra advantage.
On COMPAS we include Zafar et al.'s constrained logistic model. That paper reports an unconstrained false-positive-rate gap of 0.35 versus 0.17, and it gives no post-intervention figure for COMPAS in the file.
GFB is the post-hoc-control reference. The file reports its adjustment time of about $5\times10^{-2}$ seconds and training overhead of 1.2–1.7×, and no reported hypervolume numbers on Lacuna for Adult or COMPAS. We run it on those two datasets under the matched tuning budget.
Two ablations isolate the key ingredient. The first replaces $S$ in the projection with the identity, giving a Euclidean shift of $m$. The second applies the identical $S$-metric projection to a random-Fourier-feature model with a Laplace posterior, which tests whether the inducing-point representation matters beyond having some posterior covariance.
Why not just…?
Why not just retrain a penalized sparse GP for each metric? This is the real threat, and it may achieve slightly better accuracy because it re-fits $S$ and $Z$. Our claim is not that we beat it pointwise. We claim to come close to it at a small fraction of the cost, with metric choice deferred to deployment time. The retrained-baseline comparison settles the question. If the gap exceeds the seed-to-seed variation of the retrained models, we will report projection as a fast initializer for a short fine-tune rather than as a replacement.
Why not just post-process with barycenters or group thresholds? These methods are optimal or near-optimal for their target metric, but they read the group attribute at prediction time, which is disparate treatment in lending and hiring. Chzhen et al.'s transport solution also fragments into per-label problems under equalized odds. Our projected model is a function of $x$ alone. We will report how much of their advantage survives when that restriction is imposed on them, by running thresholds that do not depend on group.
Why not just do this projection on any linear-in-parameters model, such as random features or a network's last layer? Nothing in the algebra is GP-specific, so this is a fair challenge. The random-feature Laplace ablation answers it directly. If that model matches ours, the contribution is the covariance-metric projection itself, and we will state it that way. We expect the GP to win on groups with sparse data. There, learned inducing locations and calibrated posterior covariance should direct the constraint's cost toward uncertain regions, and fixed random features have no mechanism for this.
Why not just drop the sensitive attribute? Li et al. show that proxy features let flexible models reconstruct it; on CRIME the unaware model's 0.050 unfairness persists. Our naive baseline re-measures this on every dataset.
Proposed Experiments
We evaluate regression on CRIME, LAW and NLSY, with race or gender as the group, following Chzhen et al. We evaluate classification on Adult and COMPAS, following Zafar et al. and Sakata and Fukuchi. We use 20 random splits. The group-and-label cell statistics come from the training split only.
The metrics are RMSE or accuracy, together with the true group gaps: the demographic parity gap, the Kolmogorov–Smirnov distance between group score distributions, and the false-positive-rate and false-negative-rate gaps after thresholding. We also report the post-hoc adjustment time and the total training cost across all metrics.
The decisive check for metric generalization uses one base model per split. We project it separately under demographic parity, equal opportunity and equalized odds. We then compare each result's Pareto curve to the retrained penalized GP for that metric, including the cross-metric gaps each solution induces. The decisive check for the covariance ingredient compares the $S$-metric projection with the identity-metric projection at equal constraint satisfaction; if the $S$-metric projection does not buy lower RMSE, the core argument fails.
The decisive check for coverage uses the same data with the minority group subsampled to 5%, 10% and 20%. It tests whether stratified $Z$ reduces the accuracy lost to projection. It also tests whether that loss tracks the conditioning of $CSC^\top$ that the coreset result predicts.
To test scale we train on Adult with $M\in\{50,200,500\}$. We confirm that fitting costs match standard SVGP and that projection stays below a millisecond-scale budget comparable to GFB's reported $5\times10^{-2}$ seconds.
We state the limit plainly: first-moment constraints may leave Kolmogorov–Smirnov gaps that the barycenter method closes. Reporting the Kolmogorov–Smirnov distance alongside mean gaps measures this directly. The second-moment extension is how we would close the difference if it proves large.
References
- Kernel dependence regularizers and Gaussian processes with applications to algorithmic fairness
- Fair Classification with Efficient and Post-hoc Controllable Fairness-Accuracy Trade-off
- Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment
- Gaussian Processes for Big Data
- Scalable Variational Gaussian Process Classification
- On Coresets for Fair Regression and Individually Fair Clustering
- Fair regression with Wasserstein barycenters