Abstract
We introduce a new class of attacks on commercial-scale language models that induce jailbreaking through targeted bitwise corruptions in model parameters. Our adversary can jailbreak billion-parameter language models with fewer than 25 bit-flips in all cases—and as few as 5 in some—using up to 40× less bit-flips than existing attacks on computer vision models. Unlike prompt-based jailbreaks, our attack renders these models in memory 'uncensored' at runtime, allowing them to generate harmful responses without any input modifications. Our attack algorithm, Progressive Bit-Search, efficiently identifies target bits to flip, offering up to 20× more computational efficiency than previous methods. We demonstrate an end-to-end exploitation using Rowhammer (RH) on DRAM devices. Our findings reveal that models with less post-training alignment are more vulnerable, and certain model components, such as value projection layers, are primary targets.
Introduction
Modern Large Language Models (LLMs) are safety-aligned to prevent the generation of harmful content. While most research focuses on prompt-based jailbreaks—which use adversarial suffixes or roleplay to bypass filters—these methods are increasingly detectable. This paper explores a more fundamental threat: parameter corruption. By flipping a tiny fraction of bits in a model's weights at runtime, an attacker can permanently disable safety mechanisms without altering the input prompt.
The PRISONBREAK Attack
The primary challenge in attacking billion-parameter models is the massive search space. The authors identify the "Onion Effect": bit-flips are interdependent, meaning the impact of one flip changes the optimal location for the next. To address this, they developed the Progressive Bit-Search algorithm.
The algorithm iteratively identifies and flips one bit at a time using gradient-based importance scores. It focuses on the exponent bits of float16 weights in the value projection layers (attn.v), which were found to be disproportionately vulnerable. This targeted approach allows the attack to be 20x faster than previous search methods.
Experimental Results
The researchers evaluated PRISONBREAK across 10 diverse model families, including Llama 2, Llama 3, and Qwen 2. The attack achieved near-perfect success rates while maintaining the model's general intelligence.
Key findings include:
- High Success Rate: PRISONBREAK achieved an 80–98% Attack Success Rate (ASR).
- Minimal Utility Drop: The average accuracy drop on general tasks (measured via TinyBenchmarks) was only 2.3%, making the corruption difficult to detect through performance monitoring.
- Hardware Realism: Using Rowhammer on an NVIDIA A6000 GPU, the authors successfully jailbroke models using as few as two physical bit locations.
Mechanistic Insights
Analysis reveals that PRISONBREAK functions differently than prompt-based attacks. While prompt-based methods often try to "trick" the model into not activating its refusal circuitry, PRISONBREAK appears to leave the "refusal feature" intact but forces the model to prioritize the harmful response pathway. Furthermore, models aligned with Reinforcement Learning from Human Feedback (RLHF) proved more resilient than those using only Supervised Fine-Tuning (SFT), requiring more bit-flips to break.
Conclusion
PRISONBREAK demonstrates that the safety alignment of LLMs is fragile at the hardware level. The ability to jailbreak massive models with fewer than 25 bit-flips underscores a critical vulnerability in the current AI ecosystem. Future safety research must move beyond prompt-level defenses to consider the robustness of model parameters against physical and software-induced bit-flip attacks.
Summary updated