Abstract
As deep learning is now used in many real-world applications, research has focused increasingly on the privacy of deep learning models and how to prevent attackers from obtaining sensitive information about the training data. However, image-text models like CLIP have not yet been looked at in the context of privacy attacks. While membership inference attacks aim to tell whether a specific data point was used for training, we introduce a new type of privacy attack, named identity inference attack (IDIA), designed for multi-modal image-text models like CLIP. Using IDIAs, an attacker can reveal whether a particular person was part of the training data by querying the model in a black-box fashion with different images of the same person. Letting the model choose from a wide variety of possible text labels, the attacker can probe the model whether it recognizes the person and, therefore, was used for training. Through several experiments on CLIP, we show that the attacker can identify individuals used for training with very high accuracy and that the model learns to connect the names with the depicted people. Our experiments show that a multi-modal image-text model indeed leaks sensitive information about its training data and, therefore, should be handled with care.
Introduction
Large-scale vision-language models like CLIP have revolutionized how AI understands the relationship between images and text. By training on hundreds of millions of image-text pairs scraped from the web, these models achieve incredible zero-shot capabilities. However, this scale introduces a hidden cost: privacy.
Most privacy research in AI focuses on Membership Inference Attacks (MIAs), which ask: "Was this specific image used for training?" However, for individuals, the more pressing question is: "Was my identity used for training?" Large models are often trained on unstructured data scraped from the internet without explicit consent, frequently containing sensitive associations between names and faces. If a model memorizes these, it can be exploited to leak personal information. This paper introduces the Identity Inference Attack (IDIA) to assess these risks in multi-modal models.
Methodology
The researchers propose the Identity Inference Attack (IDIA) as a closed-box audit—meaning it only requires the ability to query the model and observe its outputs, rather than accessing internal weights.
The attack follows a structured process:
- Input: The attacker provides a few images of a person and their real name.
- Candidate Generation: They create a pool of 1,000 culturally similar "distractor" names.
- Prompt Engineering: Multiple templates (e.g., "a photo of [NAME]", "actor [NAME]") are used to query the model.
- Zero-Shot Classification: CLIP is asked to choose the best-matching name for the face from the pool of 1,000 candidates.
- Decision: If the model consistently picks the correct name across different images and prompts more often than a defined threshold, the attack concludes the person was in the training set.
Key Results and Insights
The study utilized datasets like LAION-400M and Conceptual Captions 3M (CC3M) to validate the attack. The results highlight several critical privacy concerns:
- High Precision: On models trained on LAION-400M, the IDIA achieved over 70% accuracy in identifying individuals who appeared fewer than 300 times. In controlled CC3M experiments, the attack reached a True Positive Rate (TPR) of over 95% with a False Positive Rate (FPR) below 1.8%.
- Memorization is "Cheap": A person doesn't need to be a global celebrity to be memorized. Appearing just 10 times in a dataset of nearly 3 million was enough for the model to "know" the person's face with high confidence.
- Scaling Effects: As models grow larger and more capable (e.g., moving from ViT-B/32 to ViT-L/14), they become more susceptible to IDIA. The ViT-L/14 model showed an 11.75% increase in TPR, suggesting that model scaling exacerbates privacy leakage.
Conclusion
This research serves as a wake-up call for the AI community. As we scale models to achieve emergent abilities, we must be equally diligent in developing emergent privacy protections. While masking names in training data helps, it is not a complete solution, as models can learn identities through other contextual attributes. The next challenge lies in developing reliable "machine unlearning" techniques to remove identities without the prohibitive cost of retraining models from scratch.
Summary updated