Imagine you need a vision-language model to completely forget a specific person to comply with privacy regulations. You apply a standard text-based unlearning technique, and it appears successful: when prompted with the person's name, the model no longer outputs their private biographical details.
However, a critical vulnerability remains. If you show the model a photograph of that person's face, it might seamlessly output the exact private information you thought you had erased.
This phenomenon is known as the modality gap or cross-modal asymmetry in machine unlearning. As Zheyuan Liu et al. demonstrated in Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench, knowledge in multimodal foundation models is deeply interconnected across different input types. Erasing a concept from the textual domain does not guarantee its removal from the visual domain, creating a persistent challenge for comprehensive data privacy.
Why Cross-Modal Knowledge Persists
To understand why single-modality unlearning fails, we have to look at the underlying architecture of multimodal models. Systems like Vision-Language Models (VLMs) or contrastive dual-encoders (like CLIP) typically map independent single-modality encoders into a shared latent space via an alignment projector.
When unlearning algorithms are applied superficially—such as modifying only a single modality's pathway or updating only the final layers of a network—they often merely disrupt the alignment bridge between the modalities rather than deleting the underlying concept from the weights.
As Jiali Cheng et al. point out in MultiDelete for Multimodal Machine Unlearning, standard unlearning techniques fail to account for the deep cross-modality interactions embedded within the core representations. If an algorithm only breaks the text-to-image projection layer, the internal representations of the visual encoder remain completely intact. The model can no longer link a text name to an object, but it still retains the ability to perform image-to-image retrieval or similarity matching for the supposedly forgotten concept.
Exploiting the Asymmetry: Multi-Modal Attacks
This lack of synchronization creates a severe security vulnerability. Because a forbidden concept is suppressed in one pathway but remains accessible through another, attackers can bypass safety filters by using multi-modal inputs to reactivate internal representations that were supposedly erased.
In Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearning, Renyang Liu et al. introduced the RECALL attack framework. They demonstrated that by combining a standard text prompt with a specially optimized image prompt, an attacker can force a diffusion model to regenerate "erased" content (such as copyrighted artistic styles or NSFW content) with high visual quality. This exploit succeeds precisely because current unlearning methods lack multi-modal awareness and fail to synchronize concept erasure across all input channels simultaneously.
[Text Prompt: "Van Gogh Style"] ───> (Suppressed / Unlearned) ───┐
├─> [Harmful/Copyrighted Output]
[Optimized Image Prompt] ───> (Active Visual Pathway) ───┘
Mechanisms for True Cross-Modal Erasure
To achieve robust unlearning, researchers are shifting away from isolated feature modification toward strategies that explicitly account for cross-modal dependencies.
Joint Projector-Backbone Tuning
Focusing updates on a single component leaves cross-modal associations intact. In Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset, Yingzi Ma argues that successful unlearning necessitates the simultaneous adjustment of both the alignment projector and the primary processing backbone. This coordinated approach ensures that sensitive information cannot remain embedded in the unadjusted layers of the vision-language bridge.
Collaborative Multi-Modal Supervision
Instead of relying on text-only prompts to guide the forgetting process, newer frameworks introduce multi-modal supervision during unlearning. For example, Feiran Li et al. developed the Co-Erasing framework in One Image is Worth a Thousand Words: A Usability Preservable Text-Image Collaborative Erasing Framework. By using both text prompts and the model's own generated images as visual references for what not to create, the framework overcomes the descriptive limitations of text-only prompts, achieving significantly lower attack success rates while maintaining the model's original generation quality in unrelated domains.
Modality-Aware Parameter Isolation
Another compelling direction involves identifying and isolating the specific parameters responsible for visual associations versus factual knowledge. Zheyuan Liu et al. proposed the MANU (Modality-Aware Neuron Unlearning) framework in Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Models. By identifying and pruning specific neurons that are disproportionately influential to the target "forget set" across both modalities, they successfully mitigate imbalanced unlearning.
Ultimately, these developments show that forgetting in a multimodal world cannot be achieved by treating text and images as isolated inputs. True unlearning requires modifying the shared latent spaces and overlapping neural circuits where cross-modal concepts actually live.