Abstract: The growing use of machine learning models in consequential settings has highlighted an important and seemingly irreconcilable tension between transparency and vulnerability to gaming. While this has sparked sizable debate in legal literature, there has been comparatively less technical study of this contention. In this work, we propose a clean-cut formulation of this tension and a way to make the tradeoff between transparency and gaming. We identify the source of gaming as being points close to the decision boundary of the model. And we initiate an investigation on how to provide example-based explanations that are expansive and yet consistent with a version space that is sufficiently uncertain with respect to the boundary points' labels. Finally, we furnish our theoretical results with empirical investigations of this tradeoff on real-world datasets.
Introduction: The Transparency-Gaming Tradeoff
As machine learning models increasingly govern consequential decisions—such as loan approvals or hiring—regulatory frameworks like the GDPR have introduced a "Right to Explanation." However, this demand for transparency creates a significant security risk: gaming. If a financial institution reveals exactly why a loan was denied by providing representative examples, a strategic user can aggregate these explanations to reconstruct the model’s decision boundary. Once the boundary is known, the user can strategically manipulate their features to cross the threshold without improving their underlying qualifications.
This paper, presented at AISTATS 2022, provides a rigorous technical framework to navigate this conflict. The authors propose margin-distancing, a method that preserves model security by withholding explanations for data points located near the decision boundary.
Methodology: Margin-Distancing and Version Spaces
The core of the proposed approach relies on the concept of a Version Space ($H_C$). This space represents the set of all possible models that remain consistent with the explanations an organization has publicly released. When the version space is small, a user can predict the model's behavior with high certainty, making the system vulnerable to gaming.
To mitigate this, the authors introduce margin-distancing. Instead of releasing a random or representative sample of points, the organization only releases examples that have a high "margin"—those far from the decision boundary. By removing "borderline" examples, the organization creates "strategic ambiguity," effectively enlarging the version space and making it harder for a user to pin down the exact boundary.
Empirical Evaluation and Key Results
The authors validated margin-distancing using real-world datasets, including the Credit Card Default dataset (for linear models) and the GiveMeSomeCredit dataset (for Neural Networks). They focused on "mutable" features—attributes a person could realistically change, such as debt ratio—to simulate realistic gaming scenarios.
Key Findings:
- Effectiveness in Linear Models: In experiments with Logistic Regression and SVMs, removing even a small percentage of near-boundary explanations caused boundary certainty to drop precipitously. This suggests that gaming risks can be neutralized with minimal impact on overall transparency.
- Neural Network Robustness: While the relationship was less smooth for Multi-Layer Perceptrons (MLPs), the downward trend in certainty remained clear. Simulating a user's lack of knowledge regarding model initialization further reduced vulnerability.
- The Failure of Data Augmentation: Interestingly, the study found that providing more data (even full regions of the feature space) almost always increased gaming risk. "Perfect" transparency is often the least secure.
Discussion: Insights and Limitations
One of the paper's most significant theoretical contributions is the proof that for linear models under specific conditions, margin-distancing monotonically reduces gaming risk. While this monotonicity can be "fragile" if the feature space is oddly shaped, the empirical results suggest the trend holds in practice.
However, the authors acknowledge a critical fairness concern: margin-distancing disparately affects individuals near the decision boundary. These are often the people who need explanations most to understand how to improve their standing, yet their explanations are the ones suppressed to protect the system. Future work is required to address this ethical tension and to explore whether similar distancing logic can be applied to global explanation methods like feature importance lists.
Summary updated