TL;DR
This paper introduces VGID, a novel framework for "unlearning" sensitive information from multimodal large language models (MLLMs). VGID uses a combination of visual noise and textual instructions to guide a student model to forget specific knowledge while retaining general capabilities, outperforming existing methods in both unlearning effectiveness and robustness.
What Problem Does This Paper Address?
Large multimodal models (MLLMs) are powerful tools for tasks involving both vision and language. However, their training on vast internet datasets means they can inadvertently memorize and potentially expose sensitive or private information. Machine Unlearning (MU) aims to remove this undesirable knowledge without retraining the entire model. While existing MU methods exist, they struggle with MLLMs due to the complexity of multimodal inputs. Text-only unlearning methods are insufficient because visual cues can override textual instructions, leading to privacy leaks. Furthermore, training-free methods, while preserving utility, don't truly remove knowledge at the parameter level and are vulnerable to attacks. This paper addresses the challenge of effectively unlearning in MLLMs by developing a method that balances unlearning effectiveness, model utility, and robustness against adversarial attempts.
Methods
The core of the proposed solution is Visual-Noise Guided In-Context Distillation (VGID). VGID operates as a teacher-student framework.
- Dual-Modal Intervention for Teacher Distribution: For a given "forget" data point (an image-text pair containing sensitive information), VGID first applies a visual perturbation (e.g., adding random noise) to the image and prepends a textual in-context unlearning instruction to the query. This dual-modal intervention is applied to the frozen base MLLM to generate an "unlearning-oriented teacher distribution." This distribution represents the desired "unlearned" output.
- Distillation: The student model is then trained to mimic this teacher distribution on the perturbed forget data. This guides the student model towards parameter-level unlearning.
- Utility Preservation: For "retain" data points (data that should not be unlearned), the student model is trained to match the output distribution of the original base model. This ensures that general capabilities and knowledge are preserved.
The key novelty lies in the dual-modal intervention to create a more effective teacher signal, which is then used for distillation to achieve parameter-level unlearning without needing external teacher models or explicit undesirable response annotations.
Datasets
The primary benchmark used for evaluation is MLLMU-Bench. This benchmark is specifically designed for evaluating machine unlearning in multimodal settings and includes:
- Forget sets: Containing target private knowledge to be removed.
- Test sets: For evaluating generalization to unseen private examples.
- Retain sets: For assessing the preservation of general multimodal capabilities.
MLLMU-Bench covers various task formats, including cloze-style completion, multiple-choice classification, and open-ended generation.
Key Insights and Ideas
- Visual cues are critical for MLLM unlearning: Text-only unlearning instructions are insufficient because visual inputs can strongly influence model outputs, overriding textual safety measures.
- Dual-modal intervention is more effective: Combining visual perturbation with textual unlearning instructions creates a more robust and effective teacher signal for unlearning.
- Distillation enables parameter-level unlearning: By distilling the intervention-induced teacher distribution into a student model, VGID achieves parameter-level unlearning, making it more robust than purely inference-time methods.
- Random noise is a surprisingly effective visual perturbation: Among the tested visual perturbations, random noise replacement proved most effective in removing target-sensitive visual cues while still providing a usable input for the teacher distribution.
- Strict enforcement prompts yield better results: More explicit and restrictive textual unlearning instructions lead to more reliable teacher distributions and better unlearning performance.
Key Results
VGID demonstrates strong performance across multiple MLLM architectures (Qwen2.5-VL-3B, LLaVA-1.5-7B, LLaVA-1.5-13B) and forget ratios (5%, 10%, 15%).
- Unlearning Effectiveness: In a representative setting (LLaVA-1.5-7B, 5% forget ratio), VGID reduced the forget set ROUGE-L score by 0.371 (from 0.571 to 0.200).
- Utility Preservation: This unlearning was achieved with only a 0.055 drop in the retain set ROUGE-L score (from 0.496 to 0.441).
- Robustness: VGID showed significantly better robustness against reverse-engineering attacks compared to in-context unlearning (ICU). While ICU's performance recovered substantially under attack, VGID remained largely stable.
- Trade-off: VGID achieves a better balance between unlearning completeness and model utility compared to many baseline methods, which often show imbalanced trade-offs.
Limitations and Future Work
The authors acknowledge several limitations and suggest future research directions:
- Scope of Unlearning: Current experiments focus on privacy-oriented unlearning. Future work could explore extending VGID to other scenarios like copyright protection and unsafe knowledge removal.
- Visual Perturbation Design Space: While VGID evaluates several visual perturbation strategies, the design space is vast. Developing more principled or adaptive perturbation methods could further enhance the teacher distribution quality and VGID's robustness.
Summary updated