Cybersecurity Detection Classification with Reasoning-enabled Language Models

Amol Khanna, Manu Nandan, Cristian Viorel Popa, Joan Pujol-Roig, Diana Bolocan, Laura Vasilie, Alexandru Apostu, Chase Helwig, Mihaela Gaman, Michael Brautbar, Edward Raff, Chase Midler, Sven KrasserarXiv 2026 · arXiv

TL;DR

Researchers from CrowdStrike have developed a specialized 30-billion parameter language model pipeline that uses Chain-of-Thought (CoT) reasoning to automate cybersecurity alert triage. By pairing this reasoning model with a custom-trained confidence calibrator, the system achieves 82.6% accuracy and dramatically improves the detection of benign and malicious alerts at high-confidence thresholds, outperforming even massive frontier models like Claude Opus.

What Problem Does This Paper Address?

Security Operations Centers (SOCs) are drowning in data. Modern corporate security software generates an unrelenting stream of alerts, leading to severe "alert fatigue." Because human analysts spend up to an hour investigating a single detection, labor—not alert volume—becomes the ultimate bottleneck in cybersecurity defense.

While large language models (LLMs) have been proposed to automate this triage, existing approaches typically force the model to output a direct label (e.g., "malicious" or "benign") without explaining its work. This lacks transparency and fails to leverage the inference-time reasoning capabilities of modern LLMs. Furthermore, automated triage systems require highly accurate confidence scores to safely auto-close benign alerts or prioritize critical threats. As this paper reveals, introducing reasoning actually breaks an LLM's native confidence metrics, creating a unique technical challenge.

Methods

The authors design a sophisticated four-stage post-training pipeline using a 30B parameter model (Nemotron-3-Nano) as the core classifier:

  1. Prompt Optimization: They utilize the GEPA optimizer within the DSPy framework, augmented by a custom "LLM-as-a-judge" metric. This prevents the model from converging on brittle, decision-tree-style rules (e.g., "if prevalence > 70, output benign") and instead encourages nuanced, multi-field reasoning.
  2. Self-Training (AdaSTaR): The model iteratively generates reasoning traces on difficult examples. Incorrect samples are "rationalized" via a specialized system prompt to create high-quality independent analyses for fine-tuning.
  3. Reinforcement Learning with Verifiable Rewards (RLVR): Using the GRPO algorithm, the model is rewarded if it follows the strict formatting tags (<think>...</think>) and outputs the correct final classification label.
  4. Confidence Calibration: This is the key architectural innovation. Because a Chain-of-Thought model convinces itself of the answer during its internal monologue, the final label token is always output with near-100% probability (pathological overconfidence). To fix this, the authors train a separate 30B calibrator model tasked solely with reading the detection, the reasoning trace, and the label, and predicting a single-token verdict: correct or incorrect.

Datasets

The system was trained and evaluated on a massive, real-world dataset of human-labeled Windows endpoint sensor detections from a major cybersecurity company (CrowdStrike).

  • Training set: 388,336 detections (collected over 8 weeks)
  • Validation set: 59,162 detections (collected over 2 weeks)
  • Test set: 42,686 detections (collected over 1 week)

The data includes complex telemetry such as process lineage (command lines, file paths of the process, parent, and grandparent), local/global prevalence, and MITRE ATT&CK mappings.

Key Insights and Ideas

  • Chain-of-Thought Destroys Calibration: When an LLM uses CoT, its next-token distribution for the final answer collapses into certainty. A separate judge model is not just helpful—it is a strict prerequisite for high-precision automation. Without it, high-confidence recall completely collapses to zero.
  • Domain Specialization Beats Scale: A targeted, fine-tuned 30B model significantly outperforms massive, general-purpose frontier models. When tested zero-shot, frontier models like Claude Opus exhibited severe systematic biases, frequently over-predicting alerts as "benign" and yielding dangerously low recall on actual malicious threats.
  • RL and Self-Training are Complementary: While reinforcement learning (GRPO) provides a massive boost in accuracy and makes the model's reasoning more succinct, combining it with an initial self-training phase yields the most robust and stable performance.
Figure 1: Self-training dynamics across AdaSTaR iterations. (a) Cross-entropy training loss over cumulative steps drops sharply as the model adapts to the output format and stays stable (no spikes) across iterations. (b) Stochastic (temperature 1.0) validation accuracy against the GEPA-only prompt baseline (72.0%, dashed). Accuracy is below the baseline for the first several iterations— reflecting early training instability and overfitting before the model adapts—then surpasses it and peaks at 81.7%. (c) Prediction consistency on a randomly selected 2,048-example validation subset: the always-incorrect fraction falls from 39.5% to 5.4% while the always-correct fraction grows.

Key Results

The full pipeline achieved an overall policy accuracy of 82.6%.

The true value of the system shines at the high-confidence operating points required for safe SOC automation:

  • Benign Alert Auto-Closing: At a strict 98% precision target, the proposed system achieved 64.8% recall—a massive 43.0 percentage point increase over a standard direct-label SFT baseline (which only managed 21.8%).
  • Malicious Alert Prioritization: At a 99% precision target, the system achieved 53.0% recall, outperforming the baseline's 34.7% recall by 18.3 percentage points.

Limitations and Future Work

Despite the impressive gains, the authors highlight a few key limitations:

  • Distribution Sensitivity: The high-precision operating point for benign alerts experienced a drift from 98.0% on the validation set to 90.8% on the test set, underscoring the need for continuous monitoring and periodic recalibration in production environments.
  • Scope Constraints: The current task is strictly binary (malicious vs. benign) and restricted to Windows endpoint telemetry.

Future work will focus on expanding the framework to handle multi-class triage, incorporating additional sensor platforms (such as cloud, network, and Linux logs), and developing tighter controls to mitigate validation-to-test precision gaps.

Summary updated

Abstract

A major issue in Security Operations Centers (SOCs) is alert fatigue, as the number of detections reported is more than staff can triage in a given day. Prior work prompts or fine-tunes large language models (LLMs) to emit a triage label directly, but does not train them to reason about whether a detection is a genuine threat. We train a chain-of-thought (CoT) reasoning-enabled triage classifier on real, human-labeled Windows endpoint detections by combining automated prompt optimization, self-training, and reinforcement learning with verifiable rewards. We find that CoT reasoning also degrades the label-token probabilities that automated triage relies on, so we separately train a calibrator that reads the full reasoning trace and estimates the probability that the verdict is correct. Our system reaches 82.6% test accuracy and, at the high-confidence operating point that governs automated triage, improves benign recall by 43.0% and malicious recall by 18.3% over a direct-label LLM classifier. We further show that the trained calibrator is necessary - an untrained confidence judge collapses high-confidence recall to zero - and that a finetuned 30B model significantly outperforms frontier general-purpose models, motivating targeted training over scale.
BibTeX
@misc{khanna2026cybersecurity,
  author = {Amol Khanna and Manu Nandan and Cristian Viorel Popa and Joan Pujol-Roig and Diana Bolocan and Laura Vasilie and Alexandru Apostu and Chase Helwig and Mihaela Gaman and Michael Brautbar and Edward Raff and Chase Midler and Sven Krasser},
  title = {{Cybersecurity Detection Classification with Reasoning-enabled Language Models}},
  year = {2026},
  eprint = {2607.28460},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.28460},
}
Figure