Abstract
Given an algorithmic predictor that is accurate on some source population consisting of strategic human decision subjects, will it remain accurate if the population respond to it? In our setting, an agent or a user corresponds to a sample $(X,Y)$ drawn from a distribution $\cal{D}$ and will face a model $h$ and its classification result $h(X)$. Agents can modify $X$ to adapt to $h$, which will incur a distribution shift on $(X,Y)$. Our formulation is motivated by applications where the deployed machine learning models are subjected to human agents, and will ultimately face responsive and interactive data distributions. We formalize the discussions of the transferability of a model by studying how the performance of the model trained on the available source distribution (data) would translate to the performance on its induced domain. We provide both upper bounds for the performance gap due to the induced domain shift, as well as lower bounds for the trade-offs that a classifier has to suffer on either the source training distribution or the induced target distribution. We provide further instantiated analysis for two popular domain adaptation settings, including covariate shift and target shift.
Introduction
Traditional machine learning often operates under the assumption of a static environment, where the training data distribution matches the production environment. However, in high-stakes social domains—such as credit scoring, insurance underwriting, or hiring—the deployment of a model ($h$) itself alters the environment. Human agents, seeking favorable outcomes, strategically adapt their behavior in response to the model's criteria. This feedback loop creates an induced data distribution $D(h)$, leading to what we term Induced Risk: the expected error of a model on the distribution it has itself catalyzed.
Theoretical Framework and Methodology
The core challenge is quantifying model transferability: how much does accuracy degrade when a population responds to a deployed predictor? We bridge the gap between classical domain adaptation and performative prediction by deriving general transfer bounds using $\mathcal{H}$-divergence.
Induced Shift Models
We categorize the induced distribution shift into two primary regimes, illustrated via causal graphs:
- Covariate Shift: Agents modify their features $X$ (e.g., paying down debt) to improve their classification $h(X)$, while their underlying label $Y$ remains invariant.
- Target Shift (Replicator Dynamics): The population composition itself evolves over time based on who succeeds under the current model, often modeled through evolutionary game theory.
Transfer Bounds
We provide upper bounds on the performance gap between the source training data and the induced target data. A key finding is that transfer error is fundamentally constrained by the "manipulation budget" ($B$) available to agents. If the cost of adaptation is high, the model remains more robust. Conversely, we establish lower bounds proving that in many responsive systems, a model must inevitably trade off accuracy between the source and induced distributions.
Experimental Validation
We validated our theoretical framework using both synthetic datasets generated via Structural Equation Models (SEMs) and the real-world FICO Credit Score dataset (300,000+ records). In the FICO experiments, we simulated a multi-step resource allocation scenario where a bank's policy causes the population's credit scores to evolve over $K$ time steps.
Key Results
The empirical results demonstrate that our theoretical bounds effectively bracket the observed performance degradation. As shown in the figure below, the actual performance gap ("Diff") closely tracks the predicted Upper Bound (UB) as the population adapts over successive iterations.
Our analysis further revealed that:
- Source-Optimal Suboptimality: Models optimized solely for the training distribution often incur significant induced risk upon deployment.
- Bound Tightness: Bounds for target shift (population evolution) are generally tighter and more predictive than those for covariate shift, likely due to the inherent noise in individual strategic behavior.
Discussion and Future Work
While our framework allows developers to stress-test models before deployment using hypothetical response models, it relies on certain assumptions, such as agent rationality. In practice, human responses may be constrained by unobserved factors or irrationality.
Future research should focus on identifying the "target-optimal" model—the predictor that minimizes risk on the induced domain. This may involve "bandit feedback" mechanisms to learn from agent responses in real-time, moving beyond static bounds toward dynamic optimization in interactive environments.
Summary updated