TL;DR
This paper introduces ExtremeAIGC, a new benchmark dataset and evaluation framework designed to test the vulnerability of Large Multimodal Models (LMMs) to AI-generated extremist content. The study reveals alarming weaknesses in current LMMs, demonstrating that even advanced safety measures can be bypassed by sophisticated multimodal attacks, highlighting an urgent need for more robust defenses.
What Problem Does This Paper Address?
Large Multimodal Models (LMMs) are increasingly powerful, integrating text and image understanding for diverse applications. However, this multimodal capability also makes them vulnerable to sophisticated attacks that leverage both text and images to generate harmful content, such as extremist propaganda. Existing safety benchmarks often lack AI-generated images, diverse image generation models, and comprehensive coverage of historical events, hindering a thorough assessment of LMM vulnerabilities. This paper addresses this gap by creating a specialized benchmark to evaluate how well LMMs can resist AI-generated extremist content, particularly when combined with adversarial "jailbreaking" techniques.
Methods
The authors developed a four-stage process to construct the ExtremeAIGC dataset:
- Event and Attribute Curation: They compiled 28 major extremist events and identified key attributes (person, place, organization) for each.
- Image Generation Prompt Generation: Using GPT-4, they crafted detailed text prompts based on events, attributes, and extremist topics to guide image generation.
- Image Generation: They used state-of-the-art image generation models (FLUX, SDXL, Stable Diffusion 3) to create realistic images.
- Quality Control and Filtering: A rigorous process of automated and manual filtering was applied to ensure high-quality, relevant images.
For evaluation, they employed four jailbreaking techniques:
- Generation-Based: FigStep (embedding instructions in typographic visuals) and HADES (iteratively refining images for toxicity).
- Optimization-Based: Query Attack (iteratively querying with modified inputs) and Visual Adversarial Attack (subtle image perturbations).
These attacks were tested against six state-of-the-art LMMs, and their effectiveness was measured using Attack Success Rate (ASR).
Datasets
The paper introduces ExtremeAIGC, a novel benchmark dataset comprising 3141 high-quality images generated from 1047 text prompts, covering 28 major extremist events spanning the past 200 years. The dataset is categorized into four extremist topics: Emotional Content, Misinformation, Recruitment, and Attack Planning.
Existing datasets for LMM safety evaluation are also discussed and compared, highlighting their limitations in AI-generated imagery and multimodal attack coverage.
Key Insights and Ideas
- Multimodal Attacks are Highly Effective: The combination of AI-generated images and text significantly amplifies the success rate of jailbreaking attacks compared to text-only prompts, even when the text prompt alone would be refused.
- Visuals are a Critical Vulnerability Vector: LMMs are more susceptible to adversarial attacks when visual information is involved, as these attacks can bypass text-based safety filters by embedding harmful instructions or context within images.
- Generation-Based Attacks Show High Efficacy: Techniques like FigStep and HADES, which embed harmful instructions directly into images, generally achieved higher Attack Success Rates (ASR) across most tested LMMs.
- Realistic Images Increase Risk: More realistic and semantically aligned AI-generated images tend to lead to higher vulnerability, suggesting that the quality and fidelity of generated visuals play a crucial role in attack success.
- Current Safety Measures are Insufficient: Even advanced LMMs with sophisticated safety mechanisms struggle to defend against these multimodal adversarial attacks, indicating a significant gap in current AI safety research.
Key Results
- All four jailbreaking techniques tested could significantly compromise the safety of the six target LMMs.
- Generation-based techniques (FigStep and HADES) generally achieved the highest ASR values.
- LLaVA-1.5-7B and Qwen-2-7B were identified as the most vulnerable models, with ASRs exceeding 72% for Query Attack.
- InternLM-XComposer2-VL-7B showed relatively stronger resistance but was still susceptible.
- The baseline ASR without jailbreaking was significantly lower, confirming that the models have a reasonable level of robustness under normal conditions.
- Heatmaps revealed that adversarial perturbations impact specific regions of the LMM's visual processing pipeline, with Flux-generated images showing higher attack efficacy.
Limitations and Future Work
- Scope of Content: The ExtremeAIGC dataset focuses specifically on extremist content and does not encompass the full spectrum of potential harmful content (e.g., misinformation on other topics, biased content, personally identifiable information).
- Adversarial Attack Subset: The jailbreaking techniques explored represent a subset of possible adversarial attacks; future attacks may employ different strategies.
- Arms Race: Defenses developed against current attacks might be bypassed by future, more sophisticated attacks.
- Future Work: The authors suggest that the pipeline is modular and can be extended to other harmful domains (hate speech, adult content, PII leakage). Future iterations of ExtremeAIGC will aim to expand the dataset to these areas for broader coverage of multimodal safety risks. They also emphasize the need for more robust safety measures and adversarial training paradigms.
Summary updated