Making Every Step Effective: Jailbreaking Large Vision-Language Models Through Hierarchical KV Equalization

Shuyang Hao, Yiwei Wang, Bryan Hooi, Jun Liu, Muhao Chen, Zi Huang, Yujun Cai

Abstract

In the realm of large vision-language models (LVLMs), adversarial jailbreak attacks serve as a red-teaming approach to identify safety vulnerabilities. We identify a critical limitation: not every adversarial optimization step leads to a positive outcome, and indiscriminately accepting optimization results at each step may reduce the overall attack success rate. To address this, we introduce HKVE (Hierarchical Key-Value Equalization), a framework that selectively accepts gradient optimization results based on the distribution of attention scores across different layers. Extensive experiments demonstrate HKVE’s effectiveness, achieving attack success rates of 75.08% on MiniGPT4, 85.84% on LLaVA, and 81.00% on Qwen-VL, substantially outperforming existing methods.

Introduction

Large Vision-Language Models (LVLMs) like LLaVA and GPT-4o have shown remarkable capabilities, but they remain vulnerable to "jailbreaking"—attacks where malicious image-text pairs trick the model into bypassing safety filters. Traditional white-box attacks use gradient descent to optimize an image until it triggers a harmful response. However, this process is often inefficient because it treats every optimization step as equally beneficial.

The Challenge: Finding the "Goldilocks Zone"

The researchers observed that standard gradient-based attacks often fluctuate between two undesirable states. If an attack step causes the model to focus too heavily on harmful image content, internal safety guardrails are triggered, resulting in a "Safe Response." Conversely, if the model focuses too much on the bypass prompts, it may ignore the safety filter but produce an "Uninformative Response."

The concept of attention equalization: balancing attention between prompts and information patches is key to a successful jailbreak.

The key to a successful jailbreak lies in finding an equilibrium—a state where attention is balanced enough to circumvent filters while still processing the malicious intent.

Hierarchical Key-Value Equalization (HKVE)

To address these inefficiencies, the authors propose HKVE (Hierarchical Key-Value Equalization). This framework introduces a selective acceptance mechanism for optimization steps based on internal attention distributions.

The HKVE framework: Gradient-based optimization combined with Key-Value Equalization to ensure every step is effective.

1. The "Vision Sink" Discovery

A pivotal finding of the study is that adversarial information flow is predominantly concentrated in the first two layers of the transformer stack. These "vision sinks" handle the bulk of the initial image processing. By focusing optimization efforts here, HKVE achieves a "hierarchical" efficiency that avoids the computational cost of analyzing the entire model.

2. KV Equalization and Selective Acceptance

Instead of blindly minimizing a loss function, HKVE monitors the standard deviation of attention scores in these initial layers. A lower standard deviation indicates that attention is more "equalized" across the image patches. At each step $k$, the algorithm generates an intermediate image and compares its attention distribution to the previous state. If the new image is more equalized, it is accepted with higher weight; if the step skews attention further, its contribution is minimized.

Experimental Results

The study evaluated HKVE using MM-SafetyBench, covering 13 prohibited scenarios across several prominent LVLMs.

  • Superior Success Rates: HKVE achieved Attack Success Rates (ASR) of 75.08% on MiniGPT4, 85.84% on LLaVA, and 81.00% on Qwen-VL. This represents a significant improvement of 20% to 26% over previous state-of-the-art methods like VAE and BAP.
  • Enhanced Efficiency: By filtering out counter-productive optimization steps, HKVE reached optimal convergence in just 1,500 steps, compared to the 3,500–4,000 steps required by standard attacks—a 60% reduction in compute.
  • Cross-Model Transferability: Adversarial images optimized on open-source models proved effective against proprietary systems, achieving an average ASR of 50.56% on Gemini-1.5, 58.33% on Claude-3.5, and 47.78% on GPT-4o.
  • Robustness Against Defenses: The method remained effective even when tested against advanced defense mechanisms like ECSO and AdaShield, showing only a marginal performance dip.

Conclusion

HKVE demonstrates that "balanced attention" is a critical factor in bypassing LVLM safety filters. By making every optimization step effective through hierarchical KV equalization, the framework not only increases attack potency but also significantly reduces computational overhead. These findings provide a new perspective for red-teaming and suggest that future defense mechanisms should be more "attention-aware" to remain robust.

Summary updated

Abstract

In the realm of large vision-language models (LVLMs), adversarial jailbreak attacks serve as a red-teaming approach to identify safety vulnerabilities of these models and their associated defense mechanisms. However, we identify a critical limitation: not every adversarial optimization step leads to a positive outcome, and indiscriminately accepting optimization results at each step may reduce the overall attack success rate. To address this challenge, we introduce HKVE (Hierarchical Key-Value Equalization), an innovative jailbreaking framework that selectively accepts gradient optimization results based on the distribution of attention scores across different layers, ensuring that every optimization step positively contributes to the attack. Extensive experiments demonstrate HKVE’s significant effectiveness, achieving attack success rates of 75.08% on MiniGPT4, 85.84% on LLaVA and 81.00% on Qwen-VL, substantially outperforming existing methods by margins of 20.43%, 21.01% and 26.43% respectively. Furthermore, making every step effective not only leads to an increase in attack success rate but also allows for a reduction in the number of iterations, thereby lowering computational costs.
BibTeX
@inproceedings{hao2025making,
  author = {Shuyang Hao and Yiwei Wang and Bryan Hooi and Jun Liu and Muhao Chen and Zi Huang and Yujun Cai},
  title = {{Making Every Step Effective: Jailbreaking Large Vision-Language Models Through Hierarchical KV Equalization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=EWJJ0Gmv9p},
}
Figure