Can LLM Safety Be Ensured by Constraining Parameter Regions?

Zongmin Li, Jian Su, Farah Benamara, Aixin SunarXiv 2026 · arXiv

TL;DR

This paper challenges the popular assumption that Large Language Models (LLMs) contain localized, stable "safety regions" within their parameters. By systematically evaluating four state-of-the-art identification methods, the authors demonstrate that discovered safety regions vary wildly depending on the dataset used to find them and fail to converge to a stable parameter set, especially when disentangled from general utility capabilities.

What Problem Does This Paper Address?

As LLMs are deployed globally, keeping them safe from jailbreaks and preventing "safety degradation" during downstream fine-tuning is a massive challenge. A promising line of recent research suggests that LLMs possess localized "safety regions"—specific subsets of weights, neurons, or layers uniquely responsible for safety behaviors. If we can locate these regions, we could theoretically freeze them during fine-tuning to preserve safety while adapting the rest of the model to new tasks.

However, the authors point out a critical gap: existing literature lacks a shared understanding of what properties a safety region must possess to be reliable. They argue a true safety region must satisfy three criteria:

  1. Reliability: Modifying it directly changes safety behaviors.
  2. Locality: Modifying it does not harm non-safety (utility) capabilities.
  3. Convergent Identifiability: The region must be an intrinsic property of the model, meaning different safety datasets should point to the exact same parameter region.

While prior work has looked into Reliability and Locality, this paper is the first to rigorously test Convergent Identifiability.

Methods

The authors systematically evaluate four recent safety region identification methods that span different granularities of the transformer architecture:

  • SNIP & Wanda (Parameter-level): Identifies individual scalar weights within weight matrices based on loss or output changes.
  • SafeNeuron (Neuron-level): Identifies specific rows or columns of weight matrices that are consistently activated by harmful queries.
  • NLSR (LoRA weight-level): Focuses on safety-critical parameters within low-rank adaptation matrices.
  • SafeLayer (Layer-level): Identifies specific consecutive intermediate layers where the model differentiates between harmful and benign queries.

The framework evaluates these methods across four major LLM families (Llama-2, Llama-3, Mistral, and Qwen) using the Intersection over Union (IoU) metric to measure how much the identified safety regions overlap across different datasets. Crucially, they also introduce Iso-Utility IoU, which measures the overlap of safety regions after subtracting parameters that contribute to general utility (benign queries), isolating the parameters that solely govern safety.

Overview of current safety region identification methods spanning different granularities.

Datasets

The study utilizes several public safety and utility benchmarks:

  • Multi-Category Safety Datasets: 10 distinct datasets sampled from the training splits of PKU-SafeRLHF-QA and PKU-SafeRLHF-30K, covering diverse harm categories.
  • Single-Category Safety Datasets: Subsets restricted to single harm types (e.g., Cybercrime, Drugs, Violence) to evaluate the impact of dataset semantics.
  • Baseline Safety Datasets ($D_0$): AdvBench and the original splits from the evaluated papers.
  • Utility Dataset ($D_u$): Alpaca-Cleaned, filtered to ensure no harmful queries are present, used to isolate utility-related parameters.

Key Insights and Ideas

  • The Mirage of Dataset-Agnostic Safety Regions: Current safety localization methods do not uncover universal safety mechanisms. Instead, they pick up on dataset-specific patterns.
  • The Safety-Utility Entanglement: Safety and utility are deeply intertwined in the weight space. When general utility parameters are removed from the equation, the remaining "pure" safety parameters become highly fragmented and show almost zero consistency across datasets.
  • Coarse-Grained Replication Failures: The authors were entirely unable to replicate the layer-level localization results of SafeLayer, finding that the method's over-rejection evaluation was highly sensitive and non-reproducible using standard keyword matching or GPT-4 judges.

Key Results

  • Low to Moderate Overlap: Across 10 multi-category datasets, the standard safety regions yielded an IoU of only 0.28 to 0.72. In an extreme case, the LoRA-level method (NLSR) on Mistral-7B yielded an IoU of just 0.01, showing extreme dataset dependence.
  • The Iso-Utility Collapse: When utility parameters were removed, the overlap plummeted drastically. For instance, SafeNeuron's IoU dropped from 0.58 down to 0.14–0.19 across models.
  • Semantic Sensitivity: For parameter-level methods (SNIP/Wanda), there was a moderately strong Pearson correlation ($r \in [0.51, 0.77]$) between the semantic similarity of the datasets and the overlap of the safety regions. Neuron-level methods (SafeNeuron) showed weaker semantic correlation but still failed to converge, implying they rely on internal model patterns that are still highly unstable.
Figure 2: Utility-isolated safety region overlap analysis using SafeNeuron on Llama-3-8B-Instruct. (a) We begin with D0 and gradually add one dataset at a time, in the order from D1 to D9. Next, we isolate each identified safety region with the utility region identified by Du; (b) We begin with D9 and gradually add one dataset at a time, in the order from D8 to D0; Next, we isolate each identified safety region with the utility region identified by Du; (c) The matrix is symmetric, and each element represents the semantic cosine similarity between the centroid embeddings of two multi-category identification datasets. (d) The matrix is symmetric. Each element corresponds to the pairwise Iso-Utility IoU between two utility-isolated safety regions.

Limitations and Future Work

The authors acknowledge several limitations:

  1. Scope: The evaluation is restricted to English-dominant datasets and moderate-scale models (up to 13B parameters).
  2. Metric Narrowness: IoU is a strict set-theoretic metric; it doesn't capture softer, continuous parameter alignments.
  3. Weight Space vs. Representation Space: The study focuses entirely on static weights.

As a future direction, the authors suggest shifting focus away from static weight-space "regions" toward representation-space or activation-level analyses, exploring how safety mechanisms dynamically manifest during inference across diverse, multilingual contexts.

Summary updated

Abstract

Large language models (LLMs) are often assumed to contain ``safety regions'' -- parameter subsets whose modification directly influences safety behaviors. We conduct a systematic evaluation of four safety region identification methods spanning different parameter granularities, from individual weights to entire Transformer layers, across four families of backbone LLMs with varying sizes. Using ten safety identification datasets, we find that the identified safety regions exhibit only low to moderate overlap, as measured by IoU. The overlap drops significantly when the safety regions are further refined using utility datasets (\ie non-harmful queries). These results suggest that current techniques fail to reliably identify a stable, dataset-agnostic safety region.
BibTeX
@misc{li2026can,
  author = {Zongmin Li and Jian Su and Farah Benamara and Aixin Sun},
  title = {{Can LLM Safety Be Ensured by Constraining Parameter Regions?}},
  year = {2026},
  eprint = {2602.17696},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.17696},
}
Figure