I Prefer not to Say: Protecting User Consent in Models with Optional Personal Data

Tobias Leemann, Martin Pawelczyk, Christian Thomas Eberle, Gjergji Kasneci

TL;DR

This paper introduces Protected User Consent (PUC), a framework that prevents machine learning models from penalizing users who choose not to share optional personal data. By using a novel data augmentation technique called PUCIDA, the authors ensure that a user's decision to "prefer not to say" doesn't inadvertently become a predictive signal that leads to discriminatory outcomes.

What Problem Does This Paper Address?

In many modern automated systems—like health insurance pricing—users are asked for mandatory "base" features (e.g., age, residence) but can voluntarily provide "optional" features (e.g., data from a fitness tracker) in exchange for potential benefits.

The core problem is Availability Inference. Standard machine learning models are "greedy"; they learn from every available signal, including the absence of data. If fit individuals are more likely to share their fitness data and less fit individuals are not, a model will learn that "missing data = unfit." Consequently, a user who exercises their legal right to privacy (under regulations like GDPR or CCPA) is effectively penalized with a higher insurance premium just for staying silent. The authors argue that the decision not to share data is private information in itself and should not be used to a user's disadvantage.

Figure 1: Overview of the relevant stakeholders. We con- sider a case where users can voluntarily provide information on optional features or choose to leave them undisclosed. The goals of sharers, non-sharers, and the decision maker have to be reconciled.

Methods

The authors formalize a requirement called Availability Inference Restriction (AIR), which mandates that a model's prediction for a non-sharer must not be worse than a "base model" trained only on mandatory features. To achieve this, they propose:

  1. Protected User Consent (PUC): A theoretical optimal predictor that satisfies AIR. It provides the best possible estimate for sharers using all data, while for non-sharers, it reverts to the best estimate based strictly on base features, marginalizing out the missingness signal.
  2. PUCIDA (PUC-inducing Data Augmentation): A model-agnostic training strategy. Instead of just training on the raw data, PUCIDA creates synthetic samples. For every instance where a user did share optional data, a synthetic twin is created where that data is hidden (set to N/A), but the label remains the same.
  3. Weighting Scheme: To ensure the model doesn't just learn a new bias, PUCIDA uses a specific weighting and sampling strategy to make the label distribution for "missing" data identical to the overall label distribution. This effectively "blinds" the model to the predictive power of the missingness itself.
Figure 3: Explaining PUCIDA. Our data augmentation pro- cedure expands each instance with optional information into two samples: The original instance and a synthetic sam- ple ( +⃝). The synthetic samples retain the base features and the labels, but the information on the optional features is dropped (fitness score −→N/A). The model sees samples with the same base features with a missing value and will thus base its decision only on the base features. In this exam- ple, given the base features (“NSW”, basic) and no optional statements, the model would estimate the costs to be 24k$, which is the dataset average conditioned on these values.

Datasets

The authors evaluated their approach on eight real-world datasets and two synthetic benchmarks:

  • Classification: Diabetes (diab), COMPAS (recidivism), UCI Adult (income), Water treatment (water), and Horse colic (colic).
  • Regression: California Housing (calif), ACSIncome (income), and a Health Insurance dataset (insurance).
  • Synthetic Benchmarks: Used to verify theoretical convergence rates and the model-agnostic nature of the approach.

Key Insights and Ideas

  • Privacy vs. Performance is not a Zero-Sum Game: The authors prove that a decision-maker can still benefit from the extra data provided by sharers (improving overall accuracy) without systematically disadvantaging those who choose not to share.
  • "Fairness through Unawareness" Fails: Simply dropping the optional feature for non-sharers isn't enough. If the fact that a feature is missing is correlated with the outcome, complex models (like Neural Networks or Gradient Boosted Trees) will find a way to exploit that correlation.
  • Model Agnosticism: Because PUCIDA is a data-level intervention, it works across various architectures, from simple Decision Trees to Multi-layer Perceptrons.

Key Results

  • Quantifying the Penalty: Without protection, non-sharers faced significant disadvantages. In the income and compas datasets, full-feature models assigned outcomes that were up to 18% less favorable for non-sharers compared to a base model.
  • Eliminating the Gap: Applying PUCIDA virtually eliminated this "fairness gap" across all tested model types (Random Forests, MLPs, etc.), bringing the PUC-Gap close to zero.
  • Convergence: The authors proved and empirically verified that the error of a PUC-compliant model trained with PUCIDA converges to the theoretical optimum at a rate of $O(1/N)$.
Figure 4: PUCIDA is model-agnostic. The PUC-gaps are close to zero when applying our technique across a variety of common models on the simulated dataset.

Limitations and Future Work

  • Strategic Behavior: The current work assumes users are either purely strategic (sharing only if it helps them) or purely privacy-motivated. Real-world behavior is often a messy mix of both, which could be modeled in future work.
  • Long-term Dynamics: The paper focuses on a static snapshot. Future research could explore how PUC-compliant models affect user behavior over time—for instance, does it encourage more people to share data if they know they won't be penalized for staying silent?
  • Computational Overhead: While model-agnostic, the "exhaustive augmentation" variant of PUCIDA can increase dataset size, which might be a concern for extremely large-scale applications.

Summary updated

Abstract

We examine machine learning models in a setup where individuals have the choice to share optional personal information with a decision-making system, as seen in modern insurance pricing models. Some users consent to their data being used whereas others object and keep their data undisclosed. In this work, we show that the decision not to share data can be considered as information in itself that should be protected to respect users' privacy. This observation raises the overlooked problem of how to ensure that users who protect their personal data do not suffer any disadvantages as a result. To address this problem, we formalize protection requirements for models which only use the information for which active user consent was obtained. This excludes implicit information contained in the decision to share data or not. We offer the first solution to this problem by proposing the notion of Protected User Consent (PUC), which we prove to be loss-optimal under our protection requirement. We observe that privacy and performance are not fundamentally at odds with each other and that it is possible for a decision maker to benefit from additional data while respecting users' consent. To learn PUC-compliant models, we devise a model-agnostic data augmentation strategy with finite sample convergence guarantees. Finally, we analyze the implications of PUC on challenging real datasets, tasks, and models.
BibTeX
@inproceedings{leemann2024i,
  author = {Tobias Leemann and Martin Pawelczyk and Christian Thomas Eberle and Gjergji Kasneci},
  title = {{I Prefer not to Say: Protecting User Consent in Models with Optional Personal Data}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i19.30126},
}
Figure