Condensing Law of Security Information

Qi Tan, Tianyu Cui, Qi Li, Ke XuarXiv 2026 · arXiv

In the world of Large Language Models (LLMs), the "Scaling Law" has long been treated as gospel: more parameters plus more data equals better performance. However, a new paper titled "Condensing Law of Security Information" argues that for cybersecurity, this "bigger is better" mantra isn't just wrong—it’s dangerous.

TL;DR

The paper proposes the Condensing Law, which suggests that as LLMs grow larger, they become so good at "repairing" anomalous inputs into normal-looking data that they effectively hide security threats from themselves. This creates a U-shaped error curve where security performance peaks at an optimal model size ($N^*$) and then declines as models get larger.

What Problem Does This Paper Address?

Standard LLM training focuses on likelihood fitting—learning to predict the most probable next token based on a massive corpus of "normal" data. Security, however, lives in the "long tail" of the distribution. Threats like zero-day attacks are, by definition, anomalies.

The authors identify a fundamental conflict:

  1. Cognitive Capability: Larger models have a better "picture of the world" and are superior at inferring intent from partial or corrupted inputs (think of how a large model excels at auto-completing broken code).
  2. Security Retention: In a security context, the "corruption" is the signal. If a model is too good at "fixing" a malicious SQL injection to look like a benign query, the downstream security head never sees the threat.

The paper seeks to formalize this tradeoff and prove that there is a finite "sweet spot" for model size in security tasks.

Methods

The authors use an information-theoretic framework to analyze the flow of security information through an LLM. They divide the inference pipeline into three stages:

  • Restoration (Shallow layers): The model "repairs" the input to match its perception of the real world.
  • Representation (Middle layers): The model extracts features from this repaired input.
  • Decision (Deep layers): The model makes a security judgment.
Figure 1: Pipeline of LLM

They derive three key theorems:

  1. Theorem 1: Higher cognitive capability leads to better restoration (repair) of attacked inputs.
  2. Theorem 2: Better restoration leads to lower retention of security-relevant information.
  3. Theorem 3: Lower retention leads to higher error rates in security tasks.

By combining these, they define Usable Security Information ($U$) as the product of cognitive capability ($K$) and retention ($R$).

Datasets

The authors validated their theory using several benchmarks:

  • HttpParamsDataset: A collection of 31,067 items (benign vs. anomalous HTTP request parameters like SQL injection and XSS).
  • Pythia Suite: A set of models ranging from 70M to 6.9B parameters, all trained on the exact same data in the same order, allowing the authors to isolate the effect of model scale.
  • BigModelGym (Cybersecurity World Model Benchmark): 1,000 real-world security data chains covering code, traffic data, system logs, and agent trajectories.
  • StarCore: A specialized model fine-tuned using "state chains," a new data format designed to improve security reasoning.

Key Insights and Ideas

  • The "Repair" Trap: As models scale, they don't just get smarter; they get more "stubborn" about normalcy. A massive model sees a malicious obfuscation and "silently repairs" it into a familiar, benign format before the representation layer even processes it.
  • The Condensing Law: Unlike general language tasks where error decreases monotonically, security generalization error is U-shaped. There is a specific model size $N^*$ where the model is smart enough to understand the task but not so large that it over-corrects the signal.
Figure 2: Cognitive capability and security-information retention versus model size. Cognitive capability is upper-bounded by H(P). By Theorem 2, retention decreases affinely with capability, so against model size it is a reflected (and rescaled) copy of the capability curve.
  • Usable Information is Unimodal: The "usable" part of security information peaks at an intermediate size. Beyond this point, adding more parameters actually makes the model worse at security.
Usable security information peaks at N*, leading to a U-shaped error curve.

Key Results

  • Representation Convergence: Using the Pythia suite, the authors measured the "distance" between the internal representations of benign and malicious inputs. As model size increased, these representations moved closer together, proving that the model was losing the ability to distinguish between them.
  • Real-World Validation: Testing 12 contemporary models (including Qwen, GLM, and DeepSeek) on safety benchmarks confirmed the U-shape. For example, the StarCore-180B model outperformed its larger 320B counterpart, validating that further scaling reduced performance.
  • State Chains: The authors found that fine-tuning with "state chains" (a format that emphasizes the step-by-step logic of an attack) could shift the optimal point, allowing smaller models to outperform much larger general-purpose ones.

Limitations and Future Work

  • The Product Ansatz: The authors admit that the specific product form $U(N) = K(N) \cdot R(N)$ is an ansatz (a mathematical assumption) rather than a direct derivation from first principles, though it is strongly supported by their experimental data.
  • Task Specificity: The optimal model size $N^*$ likely varies depending on the specific security domain (e.g., malware analysis vs. network intrusion detection).
  • Mitigation Strategies: While the paper identifies the problem, it only briefly touches on how to "fix" it (e.g., via state chains). Future work could explore architectural changes that prevent shallow layers from over-repairing anomalous signals.

Summary updated

Abstract

Scaling laws indicate that the generalization error of a large language model (LLM) decreases monotonically with model size once data is sufficient, and the corollary that larger models outperform smaller ones is now widely accepted. We argue that this corollary fails in security: decisions there hinge on anomalous inputs, which are long-tailed (e.g., zero-day attacks) and therefore poorly served by likelihood fitting. Worse still, the capability brought about by scale (e.g., inferring the intended whole from a corrupted part) leads a LLM silently \emph{repair} the anomaly before any downstream security head ever sees it. We formalize this as a tradeoff between two quantities: the cognitive capability $K(W)=I(W;P)$ that a model holds about the world distribution, and the security-information retention $R(W)=\frac{I(Z;S)}{I(A;S)}$ that survives in its intermediate representation. We derive three information-theoretic bounds to connect them: an upper bound on restoration accuracy in terms of $K(W)$, an upper bound on $R(W)$ in terms of the restoration error, and a Fano lower bound on security-task error in terms of $R(W)$. These theorems approximately imply that usable security information is a product of how much security information survives and how well it can be extracted, indicating that optimal usable security information can be achieved at a certain model size. We state this as the \emph{condensing law of security information}: usable security information is unimodal in model size, so security generalization error is U-shaped and is minimized at a finite $N^{\star}$ rather than decreasing without bound. The experiments on various real-world LLMs validate the condensing law, indicating that there is an optimal model size to solve security issues.
BibTeX
@misc{tan2026condensing,
  author = {Qi Tan and Tianyu Cui and Qi Li and Ke Xu},
  title = {{Condensing Law of Security Information}},
  year = {2026},
  eprint = {2611.17918},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17918},
}
Figure