Margin-distancing for safe model explanation

Tom Yan, Chicheng Zhang

Abstract

The growing use of machine learning models in consequential settings has highlighted an important and seemingly irreconcilable tension between transparency and vulnerability to gaming. While this has sparked sizable debate in legal literature, there has been comparatively less technical study of this contention. In this work, we propose a clean-cut formulation of this tension and a way to make the tradeoff between transparency and gaming. We identify the source of gaming as being points close to the decision boundary of the model. And we initiate an investigation on how to provide example-based explanations that are expansive and yet consistent with a version space that is sufficiently uncertain with respect to the boundary points’ labels. Finally, we furnish our theoretical results with empirical investigations of this tradeoff on real-world datasets.

Introduction: The Transparency-Gaming Tension

As machine learning models increasingly automate high-stakes decisions—such as loan approvals or hiring—there is a growing legal and ethical demand for transparency. Regulations like the GDPR’s "Right to Explanation" suggest that users deserve to know the reasoning behind a model's decision. However, a fundamental tension exists between transparency and gaming. If an organization reveals too much about its model's internal logic, individuals can strategically "game" the system by misreporting data to cross a decision threshold without actually improving their underlying qualifications.

This paper addresses the technical challenge of providing "expansive" explanations (giving enough information to be useful) while ensuring the model remains "safe" (maintaining enough uncertainty to prevent successful manipulation).

Methodology: Margin-Distancing and Version Spaces

The authors focus on example-based explanations, where an organization releases a set of labeled prototypes to explain model behavior. Their core proposal is Margin-Distancing, a filtering strategy based on the following principles:

  1. The Margin Score: Every data point has a "margin," representing its distance from the model's decision boundary.
  2. The Filter: Instead of releasing a random subset of examples, the organization only releases examples where the margin is greater than a specific threshold $\alpha$. Examples near the boundary are withheld.
  3. The Version Space (VS): From the user's perspective, the released examples define a "Version Space"—the set of all possible models consistent with those examples.

By removing examples near the boundary, the Version Space remains large and diverse in that critical region. This creates "boundary uncertainty," making it risky for a user to attempt a specific manipulation because they cannot be certain it will successfully flip the model's prediction.

Visualizing the Version Space: As points near the boundary are removed (moving from left to right), the set of consistent decision boundaries (black lines) becomes more diverse, increasing the uncertainty for potential gaming.

Empirical Evaluation and Key Insights

The approach was validated using real-world datasets, including Credit Card Default data and the GiveMeCredit dataset, testing both linear models and Multi-Layer Perceptrons (MLPs).

Key Findings:

  • Gaming is a Boundary Problem: Strategic manipulation relies on knowing the exact location of the decision boundary. Keeping the area around the line "blurry" prevents gaming without hiding the entire model.
  • Monotonicity: For linear models, the relationship between the number of withheld examples and model safety is mostly monotonic. This allows engineers to use binary search to find the optimal number of explanations to withhold for a desired safety level.
  • Neural Network Robustness: While the theoretical proofs are more complex for MLPs, experiments show that margin-distancing remains effective. The inherent "model multiplicity" (the Rashomon Effect) in neural networks actually helps protect against gaming by making it harder to pin down a single exact boundary.
Empirical results showing the decrease in boundary certainty ($π(x_1, x_2)$) as a higher percentage of explanation points closest to the margin are removed across different radii (r).

Discussion: Ethics and Future Directions

While margin-distancing provides a technical solution to the gaming problem, it introduces significant ethical trade-offs. The strategy disproportionately affects individuals near the decision boundary—the very people who often need explanations most to understand how to legitimately improve their situation.

Future work is required to address cases where users have incorrect assumptions about the model class and to extend these safety concepts from example-based explanations to global explanations, such as feature importance weights or SHAP values.

Summary updated

Abstract

The growing use of machine learning models in consequential settings has highlighted an important and seemingly irreconcilable tension between transparency and vulnerability to gaming. While this has sparked sizable debate in legal literature, there has been comparatively less technical study of this contention. In this work, we propose a clean-cut formulation of this tension and a way to make the tradeoff between transparency and gaming. We identify the source of gaming as being points close to the decision boundary of the model. And we initiate an investigation on how to provide example-based explanations that are expansive and yet consistent with a version space that is sufficiently uncertain with respect to the boundary points’ labels. Finally, we furnish our theoretical results with empirical investigations of this tradeoff on real-world datasets.
BibTeX
@inproceedings{yan2022margin,
  author = {Tom Yan and Chicheng Zhang},
  title = {{Margin-distancing for safe model explanation}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
  url = {https://openreview.net/forum?id=Bqtlc9eDRx},
}
Figure