TL;DR
This paper introduces S1Mal, a malware detection framework built on a new class of "System One" decision models that provide fast, structured, and calibrated probabilities instead of slow, free-form text. By combining deep behavioral questioning with internal state embeddings, S1Mal achieves an impressive 0.9946 AUROC, offering security analysts both high-performance detection and a clear "account of evidence" for every decision.
What Problem Does This Paper Address?
Malware detection has long been a tug-of-war between speed and explainability.
- Feature-based detectors (like LightGBM) are lightning-fast and easy to calibrate for low False Positive Rates (FPR), but they are "black boxes"—they give a score without explaining why a file is malicious.
- Large Language Models (LLMs) reading decompiled code offer great explanations but are slow, expensive to run, and hard to calibrate. Furthermore, they are easily fooled by "planted text"—malicious actors can simply insert strings like "this file is safe" into a binary to flip an LLM's verdict.
The authors explore a middle ground: System One decision models. These models (like CLM-8B) don't generate text; instead, they take a textual state and a typed question (e.g., "Does this program perform process injection?") and return a probability in a single forward pass.
Methods
The researchers built a pipeline that transforms a Windows Portable Executable (PE) binary into a "canonical textual state."
- Isolated Analysis: To prevent "cheating," binaries are processed in an isolated VM using Ghidra (for decompilation) and capa (to identify capabilities).
- Shortcut Control: They carefully strip out non-behavioral "shortcuts" like timestamps, file names, and toolchain strings that might allow a model to distinguish malware from benign files without actually understanding the code.
- The Readouts: They evaluated three ways to get a score from these models:
- A0 (Verdict): A simple "Is this malware?" question.
- A1 (Atomic): Asking 32 specific behavioral questions (e.g., about keylogging or ransomware encryption) and combining the answers using Gradient-Boosted Trees.
- A2 (Embedding): Training a small "head" on the model’s internal vector representation (embedding) of the file.
- S1Mal: Their flagship ensemble that fuses the behavioral answers (A1) with the embedding score (A2) of the CLM-8B model.
Datasets
The study utilized a robust mix of data to test for both temporal drift and out-of-distribution (OOD) performance:
- BODMAS: 57,293 malware files, split temporally (training on 2019 data, testing on 2020 data).
- SOREL-20M: Used as an OOD set to see how well models transfer to different malware sources.
- Benign PE Collection: 14,534 matched benign files from common Windows software and installers.
Key Insights and Ideas
- The "Verdict Gap": Surprisingly, asking a model "Is this malware?" (A0) performs poorly (AUROC as low as 0.40). However, asking the same model 32 specific questions about behavior (A1) "recovers" the signal, boosting AUROC by over 54% on average. This suggests these models "know" the components of malice better than the concept of "malware" itself.
- Embeddings are Richer than Questions: The embedding-based readout (A2) consistently outperformed the question-based readout. This indicates that the model's internal state captures subtle patterns that aren't fully captured by the 32 behavioral questions.
- Defense Against "Planted Text": S1Mal is significantly more robust than standard LLMs. While planting "legitimacy narratives" in a file can flip a generative LLM's verdict, S1Mal’s behavioral questions are harder to steer because they ask about specific technical artifacts (imports, capabilities) rather than general intent.
Key Results
- Detection Performance: S1Mal reached an AUROC of 0.9946. At a fixed 1% FPR, it detected 93.47% of malware.
- Low FPR Excellence: In the high-stakes region of 0.1% FPR, S1Mal maintained a detection rate of 59.15%, significantly outperforming the embedding-only approach (46.59%).
- Efficiency: The authors demonstrated that these models can run efficiently on a laptop GPU. A 5-bit quantized version of the model could analyze a file in about 0.36 seconds, making it viable for real-world endpoint protection.
Limitations and Future Work
- Static Analysis Limits: The system relies on static analysis. If a file is heavily packed or encrypted such that its behavior only appears at runtime, the current pipeline won't see it. The authors suggest integrating dynamic analysis (sandboxing) in the future.
- Context Window: The models are currently limited to an 8,192-token window. While this covers most files, very large or complex programs might have their evidence truncated.
- Policy Restrictions: The authors noted that some high-end models (like TypeSafe’s Jev) have usage policies prohibiting malware analysis, highlighting a non-technical hurdle for researchers in this space.
This summary is based on "A First Look at System One Decision Models for Malware Detection" (Nov 2026).
Summary updated