Overlearning Reveals Sensitive Attributes

Congzheng Song, Vitaly Shmatikov

In the world of machine learning, we usually worry about models not learning enough (underfitting) or memorizing their training data too closely (overfitting). But a fascinating paper by Congzheng Song and Vitaly Shmatikov introduces a third, more insidious category: Overlearning.

TL;DR

"Overlearning" occurs when a deep learning model trained for a simple task (like identifying gender) implicitly learns to recognize sensitive attributes (like race or individual identity) that were never part of its objective. This paper demonstrates that these "secret" features are intrinsic to deep learning, making standard privacy-preserving techniques like censoring and model partitioning surprisingly easy to bypass.

What Problem Does This Paper Address?

The core assumption in many privacy-preserving ML frameworks is that a model only "knows" what it was trained to do. For example, if you send a facial image to a cloud-based "gender classifier," you might assume the cloud provider only learns the gender of the person in the photo.

To bolster this, researchers use model partitioning (keeping the first few layers on a user's device to "censor" sensitive data before sending it to the cloud) or adversarial censoring (training the model to explicitly "forget" certain attributes).

Song and Shmatikov argue that these protections are a house of cards. They tackle the question: Does a model trained for Task A unintentionally become an expert at Task B, and can an adversary exploit that hidden knowledge?

Methods of Exploitation

The authors explore two main ways to exploit overlearning:

  1. Inference-time Attacks: An attacker observes the internal representation (the "embedding") of a model and trains a small "attack model" to predict sensitive attributes from it.
  2. Model Re-purposing: An attacker takes a pre-trained model (e.g., a benign indoor/outdoor scene classifier) and uses its lower layers as a feature extractor to train a new, privacy-violating model (e.g., a fine-grained scene recognizer) with very little data.

Crucially, they introduce a De-censoring technique. If a model has been trained to hide a specific attribute (like race), the authors show they can use an auxiliary dataset to "transform" the censored representation back into a format that leaks the information, effectively undoing the privacy protection.

Datasets Evaluated

The study is impressively broad, covering vision, text, and tabular data to prove this is a generalized phenomenon:

  • Vision: UTKFace (gender/race), FaceScrub (gender/identity), Places365 (indoor-outdoor/fine-grained scenes), and PIPA (identifying if people appear together).
  • NLP: Twitter and Yelp (predicting age/score vs. identifying the specific author).
  • Tabular: Health (predicting mortality vs. age).

Key Insights

  • Overlearning is Intrinsic: It isn't a mistake; it's a byproduct of how deep networks handle complex data. To solve a "simple" task on complex input (like a high-res photo), the model must develop sophisticated internal features that naturally capture more information than the label requires.
  • The "Blacklist" Failure: Current censoring techniques are "blacklist-based"—you have to know what you want to hide. But overlearning captures attributes you might not even know are there.
  • Generalization to the Unseen: In one of the most striking findings, a gender classifier trained only on images of one race still learned features that allowed it to distinguish between other races it had never seen during training.
  • Lower Layers are the Culprits: The "general-purpose" features that emerge in the early layers of a network are the most susceptible to being re-purposed for sensitive tasks.

Key Results

Relative accuracy of main and sensitive tasks under different censoring defenses across multiple datasets.
  • High Leakage: Even without trying to exploit the model, the "base" representations were highly revealing. For example, a gender classifier achieved 62% accuracy in identifying race (where random guessing was 42%).
  • Censoring is Fragile: While adversarial training reduced leakage, the authors' de-censoring attack recovered much of the lost information. On the FaceScrub dataset, de-censoring boosted the attack accuracy by over 12%.
  • Re-purposing Works with Tiny Data: Using a pre-trained model as a starting point allowed attackers to build highly accurate "malicious" models using only 2% to 10% of the data required to train from scratch.

Limitations and Future Work

The authors conclude with a sobering thought for regulators: GDPR and similar privacy laws are difficult to enforce in the age of overlearning. If a model implicitly learns attributes that the developer didn't ask for (and might not even know exist), how can they "disclose the purpose" of data collection or "obtain consent"?

The paper suggests that for some complex tasks, overlearning might be impossible to prevent without destroying the model's primary utility. Future work is needed to determine if there are "white-list" approaches to training—where a model is forced to only learn what is necessary—rather than the current "blacklist" approach of trying to hide specific secrets after the fact.

Summary updated

Abstract

Overlearning" means that a model trained for a seemingly simple objective implicitly learns to recognize attributes and concepts that are (1) not part of the learning objective, and (2) sensitive from a privacy or bias perspective. For example, a binary gender classifier of facial images also learns to recognize races\textemdash even races that are not represented in the training data\textemdash and identities. We demonstrate overlearning in several vision and NLP models and analyze its harmful consequences. First, inference-time representations of an overlearned model reveal sensitive attributes of the input, breaking privacy protections such as model partitioning. Second, an overlearned model can be "re-purposed" for a different, privacy-violating task even in the absence of the original training data. We show that overlearning is intrinsic for some tasks and cannot be prevented by censoring unwanted attributes. Finally, we investigate where, when, and why overlearning happens during model training.
BibTeX
@inproceedings{song2020overlearning,
  author = {Congzheng Song and Vitaly Shmatikov},
  title = {{Overlearning Reveals Sensitive Attributes}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=7VQwzexxeZ},
}
Figure