Tracing Training Footprints: A Calibration Approach for Membership Inference Attacks Against Multimodal Large Language Models

Xiaofan Zheng, Huixuan Zhang, Xiaojun Wan2025 · DOI

TL;DR

Researchers have introduced PC-MMIA, a novel black-box membership inference attack designed specifically for Multimodal Large Language Models (MLLMs). By slightly perturbing image embeddings and measuring how much text predictions fluctuate, the method can accurately distinguish between a model that has genuinely "learned" a concept and one that has simply "memorized" a specific training sample.

What Problem Does This Paper Address?

As Multimodal Large Language Models (like GPT-4o or LLaVA) are trained on increasingly massive datasets scraped from the internet, concerns about privacy and copyright have skyrocketed. Membership Inference Attacks (MIA) are the primary tool for auditing these models; they allow researchers to determine if a specific data point (like a private photo or copyrighted text) was included in a model's training set.

However, existing MIA methods—mostly designed for text-only LLMs—struggle with multimodal data. The core issue is difficulty calibration. In a text-only model, a high-probability word might indicate memorization. In an MLLM, a word might have a high probability simply because it is "obvious" given the image (e.g., the word "apple" is easy to predict if the image shows an apple). Existing attacks often produce "false positives," misidentifying these easy, generalized samples as memorized training data.

Methods

The authors propose PC-MMIA (Perturbation-based Calibration for Multimodal MIA), which operates under a "black-box" scenario—meaning the attacker only needs the probability of the predicted tokens, not the model's internal weights.

The approach relies on a clever observation: memorized data points sit on "brittle" parts of the model's data manifold. If a model has memorized a specific image-text pair verbatim, its confidence will drop sharply if the image is slightly changed. If the model has actually learned a general concept, its confidence will remain stable even with minor noise.

The PC-MMIA workflow: 1. Bidirectional Perturbation: Small amounts of symmetric Gaussian noise are added to and subtracted from the image embeddings. 2. Likelihood Tracking: The system calculates the log-likelihood of the text response for the original image and the two "noisy" versions. 3. Calibration Weighting: It calculates the average change in log-likelihood ($\delta$). Tokens that show significant fluctuations are assigned higher weights because they are more likely to represent "training traces." 4. Final Scoring: A weighted membership score is calculated. If the score exceeds a certain threshold, the sample is flagged as a "member" of the training set.

Datasets

The researchers evaluated their method by fine-tuning two popular MLLMs (LLaVA-1.5-7b and CogVLM-17b) on specific subsets of data to create a ground truth for "members" and "non-members." They used:

  • NoCaps: A dataset for novel object captioning.
  • Flickr30k: A standard image-captioning benchmark.
  • PixMo: A diverse visual instruction dataset from the Allen Institute for AI, known for varying text lengths.
  • VizWiz-VQA: Used for additional validation in Visual Question Answering scenarios.

Key Insights and Ideas

  • The "Local Smoothness" Hypothesis: The most profound insight is that memorization induces local discontinuities. Generalization is "smooth" (robust to noise), while memorization is "jagged" (sensitive to noise). PC-MMIA essentially "pokes" the model with noise to see how jagged its response is.
  • Image as a Prefix: In multimodal contexts, the image acts as a massive prefix that sets the "difficulty" for the following text. PC-MMIA is the first method to use the image itself to calibrate that difficulty, rather than just looking at text statistics.
  • Softmax Normalization Matters: The authors found that using the relative relationship of fluctuations within a sample (via softmax) was more effective than using absolute values. This eliminates bias caused by some images being inherently more sensitive to noise than others.

Key Results

PC-MMIA consistently outperformed all existing black-box baselines across all datasets and model architectures.

  • Superior Detection: On the CogVLM-17b model using the PixMo benchmark, PC-MMIA improved the AUC (Area Under the Curve) by 4.9% and the True Positive Rate (at a 1% False Positive Rate) by 6.9% compared to the previous state-of-the-art.
  • Robustness to Length: Unlike text-only MIA, which gets much easier as text gets longer, PC-MMIA remains highly effective even on shorter responses because the image embedding provides a rich signal for calibration.
  • Stability: While other methods (like "Neighbor" attacks that swap words) are unstable across different datasets, PC-MMIA’s use of Gaussian noise on embeddings proved to be a more reliable and "semantically invariant" perturbation.

Limitations and Future Work

  • Heuristic Nature: The method relies on the heuristic assumption that Gaussian noise in the embedding space preserves semantic meaning. While empirical results support this, a formal mathematical derivation of this "local smoothness" in MLLMs remains an open challenge.
  • Closed-Source Barriers: Testing on "frontier" models like GPT-4o is difficult because their training data is a closely guarded secret, making it impossible to verify "members" vs "non-members" with 100% certainty.
  • Computational Cost: Because the method requires three forward passes (original, plus two perturbed versions), it is more computationally expensive than simple loss-based attacks. Future work could focus on making the perturbation process more efficient.

Summary updated

Abstract

With the increasing scale of training data for Multimodal Large Language Models (MLLMs) and the lack of data details, there is growing concern about privacy breaches and data security issues.Under black-box access, exploring effective Membership Inference Attacks (MIA) has garnered increasing attention.In real-world applications, where most samples are non-members, the issue of non-members being over-represented in the data manifold, leading to misclassification as member samples, becomes more prominent.This has motivated recent work to focus on developing effective difficulty calibration strategies, producing promising results.However, these methods only consider text-only input during calibration, and their effectiveness is diminished when migrated to MLLMs due to the presence of visual embeddings.To address the above problem, we propose PC-MMIA, focusing on visual instruction fine-tuning data.PC-MMIA is based on the idea that tokens located in poorly generalized local manifolds can better reflect traces of member samples that have been trained.By employing bidirectional perturbation of image embeddings to capture tokens critical to MIA and assigning them different weights, we achieve difficulty calibration.Experimental results demonstrate that our proposed method surpasses existing methods.
BibTeX
@misc{zheng2025tracing,
  author = {Xiaofan Zheng and Huixuan Zhang and Xiaojun Wan},
  title = {{Tracing Training Footprints: A Calibration Approach for Membership Inference Attacks Against Multimodal Large Language Models}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.931},
}

Loading related content...

Figure