Introduction
Standard machine learning (ML) models are designed to maximize predictive accuracy by minimizing a loss function. However, real-world applications often demand more than just accuracy; they require adherence to safety protocols, physical laws, or social fairness standards. Traditionally, enforcing such constraints required modifying model architectures, ensuring loss functions were differentiable, or complex data pre-processing—all of which limit the choice of models and often degrade performance.
This paper introduces Moving Targets (MT), a model-agnostic framework that allows researchers to inject complex, even non-differentiable constraints into any supervised learning algorithm without altering the underlying learner. By treating constraints as a separate optimization problem, MT enables "old dogs" like Random Forests or Linear Regression to learn "new tricks" like fairness and class balance.
The Moving Targets Framework
The core of the MT approach is a bi-level optimization loop that decouples the learning process from the constraint satisfaction process. It iterates between two primary stages:
- The Master Step: This acts as a "teacher." It takes the current model's predictions and the original ground-truth labels to solve a mathematical optimization problem. It finds a new set of labels—the "moving targets"—that are as close as possible to the original truth while strictly satisfying the desired constraints. Because this step is independent of the ML model, it can utilize powerful solvers like Mixed-Integer Programming to handle discrete or population-level constraints.
- The Learner Step: This acts as the "student." It receives the adjusted labels from the Master Step and performs standard supervised learning. The learner remains unaware of the constraints; it simply attempts to fit the new targets provided by the teacher.
By alternating these steps, the model's predictions gradually converge toward a state that satisfies the constraints while maintaining high fidelity to the original data patterns.
Experimental Evaluation
The researchers evaluated MT across seven datasets from the UCI Machine Learning Repository, focusing on two types of constraints:
- Balance Constraints: Applied to datasets like iris, redwine, whitewine, shuttle, and dota2 to ensure predicted classes follow a specific distribution.
- Fairness Constraints: Applied to the adult (Census Income) and crime (Communities and Crime) datasets to minimize disparate impact, measured by the DIDI indicator.
Key Results and Insights
Effective Constraint Satisfaction
MT successfully reached near-feasible states across all tested datasets. Unlike specialized regularization methods that require tedious tuning of penalty weights, MT provides a more reliable path to satisfying constraints by directly manipulating the training targets.
The "Accuracy Bonus"
One of the most surprising findings was that on the adult and crime datasets, MT actually improved predictive accuracy while satisfying fairness constraints. The authors hypothesize that the iterative label-shifting acts as a guided search, helping the model escape local optima that standard training might get stuck in.
Scalability and Performance
Despite the Master Step involving potentially NP-hard optimization, the framework proved highly scalable. For the dota2 dataset, which contains approximately 74,000 examples, the optimization step was often faster than the actual training of the neural network learner.
Limitations and Future Work
While empirically successful, the black-box nature of the "Learner" makes it difficult to provide absolute mathematical convergence guarantees. Additionally, in cases of extreme data imbalance (e.g., the shuttle dataset), forcing a balanced output naturally leads to an accuracy trade-off, as the model must ignore certain skewed patterns to remain "fair" or "balanced."
Future research will explore applying Moving Targets to semi-supervised learning environments and refining the projection techniques used in the Master Step to further improve efficiency on million-row datasets.
Summary updated