Standard machine learning training follows a simple mantra: minimize the average error across all data. While efficient, this "Empirical Risk Minimization" often creates a fairness gap, where the model performs significantly better for majority groups than for disadvantaged ones.
Instead of treating fairness as a static constraint to be satisfied after training, a growing body of research treats fairness as a dynamic control problem. By using adaptive feedback loops—adjusting weights, sampling rates, or model capacity in real-time—these methods force the optimizer to "level up" performance for underserved groups as the model learns.
Fairness as a Dynamic Cost
The most direct way to enforce fairness is to make the optimizer "pay" more for errors on disadvantaged groups. However, determining the right "price" (weight) for these errors is difficult because the model's behavior changes during training.
Researchers like Han Xu and Jiliang Tang address this through the Fair-Robust-Learning (FRL) framework. In To be Robust or to be Fair, they use a Lagrangian multiplier framework where the penalty for a specific class increases dynamically whenever its error exceeds the average.
This creates a "thermostat" effect: if a group's performance drops, the system automatically turns up the "heat" (the loss weight) for that group, forcing the model to prioritize it in the next gradient update. Similar dual-variable mechanisms are used by Rad Niazadeh and colleagues in Online Combinatorial Optimization with Group Fairness Constraints, where fairness violations are treated as real-time costs that adjust based on historical underservice.
Steering the Data Stream
If the loss function is the "price" of an error, the training data is the "fuel." Another way to achieve fairness is to adaptively change what the model sees.
Kangwook Lee and Steven Euijong Whang introduced FairBatch (Paper), which modifies the composition of minibatches during training. Instead of a uniform random sample, FairBatch adaptively increases the representation of groups currently experiencing higher error rates. This "active sampling" approach is also championed by Jacob Abernethy and Jamie Morgenstern in Active Sampling for Min-Max Fairness, where the algorithm greedily picks data from the group currently suffering the highest loss.
The intuition here is that the model's decision boundary is constantly shifting. By dynamically over-sampling the "worst-off" groups, the optimizer is forced to move the boundary in a way that specifically reduces the maximum error across all groups, achieving what is known as min-max fairness.
Modulating Optimization Dynamics
Beyond weights and data, fairness can be baked into the optimization process itself—the "speed" and "capacity" of learning.
- Learning Rate Modulation: In Fair Federated Graph Neural Networks, Jayadeva and colleagues propose throttling the learning rate for over-served groups while accelerating it for under-served ones. This ensures that the model doesn't "over-fit" to the majority before the minority groups have had a chance to influence the weights.
- Dynamic Capacity Allocation: Real-world data often has intersecting sensitive attributes (e.g., age, race, and gender). Changdi Yang and Zheng Zhan tackle this in FairSMoE by using a Sparse Mixture-of-Experts (SMoE) architecture. Their system monitors fairness metrics and dynamically allocates more "expert" neurons to attributes that are currently being treated unfairly, effectively giving the model more "brain power" to resolve complex biases.
- Attention as a Filter: Stefan Nielsen and Tan Minh Nguyen use attention mechanisms in FARE to dynamically weight samples. The model learns to "ignore" samples that contribute to bias, effectively filtering the representation space in real-time to prioritize fairness-neutral features.
Why It Works: The Phase Transition of Fairness
The power of these adaptive methods lies in their ability to handle non-stationary environments. In No-regret Algorithms for Fair Resource Allocation, Abhishek Sinha and Cameron Musco demonstrate that adaptive algorithms can achieve sublinear regret even against adversarial data distributions. They identify a "phase transition" where, once a certain fairness threshold is reached, the adaptive weights stabilize, allowing the model to maintain equity without sacrificing long-term utility.
By treating fairness as a moving target that requires constant adjustment, these frameworks move away from the "train then fix" mentality. Instead, they create models that are inherently self-correcting, ensuring that robustness and accuracy are distributed equitably across the entire population.