TL;DR
SecPatRec is a novel hybrid tool that automates the recovery of security pattern implementations from source code by combining the semantic reasoning of Large Language Models (LLMs) with the structural precision of Static Analysis. It achieves high accuracy in identifying architectural roles and their interactions while being significantly more scalable and cost-effective than pure LLM-based approaches.
What Problem Does This Paper Address?
"Security-by-design" is a fundamental principle in modern software engineering, yet verifying that a system actually adheres to its intended security architecture is notoriously difficult. Developers often use security patterns—reusable solutions to recurring security problems like authentication or rate limiting—but these patterns are abstract.
A single "role" in a pattern (e.g., a "Verification Manager") might be implemented as a specific class, a set of functions, or even a call to an external library. Furthermore, architectural documentation is frequently missing or out of sync with the actual code. Existing automated recovery methods either rely on rigid graph-matching (which fails to capture semantic meaning) or "one-shot" LLM prompts (which are expensive and limited by the "context window" of the model).
Methods
The authors propose a hybrid pipeline called SecPatRec (Security Pattern Recovery) that treats pattern recovery as a deductive task.
- ADL Generation: The system first translates human-readable pattern catalogs (containing text and UML diagrams) into a machine-readable Architecture Description Language (ADL) using a specialized generator.
- Role Detection & Grounding (The LLM Agent): Instead of feeding the entire codebase into an LLM, SecPatRec uses a ReAct-style agent. The agent iteratively reasons about the code and "acts" by using tools like semantic search (finding code by meaning) and lexical search (grep). This allows it to pinpoint which code entities implement specific pattern roles.
- Interaction Recovery (Static Analysis): Once roles are mapped to code, the system uses a Code Property Graph (CPG)—specifically the Joern platform—to trace the actual execution paths and call sequences between these entities. This ensures that the recovered behavior is based on the actual logic of the code, preventing the "hallucinations" common in LLMs.
Datasets
The researchers introduced a new, human-validated ground-truth dataset to evaluate their approach:
- 100 pattern instances across 93 open-source repositories.
- Languages covered: Java (Spring Boot) and Python (FastAPI).
- Five core security patterns: Password-based authentication, Verifiable-token authentication, Obscure-token access control, Session-based access control, and Rate limiting.
Key Insights and Ideas
- Semantic vs. Structural: The "aha moment" here is the division of labor. LLMs are excellent at understanding what a piece of code represents semantically (e.g., "this function looks like it's checking a password"), while static analysis is unbeatable at determining how those pieces are structurally connected.
- Scalability through Search: By giving the LLM "tools" to search the codebase rather than reading it all at once, the system can process repositories that are far too large for a standard prompt window.
- Cost Efficiency: The hybrid approach is dramatically cheaper. For a 1-million-token codebase, SecPatRec is between 6x and 60x less expensive than a one-shot LLM baseline.
- The "Framework Magic" Gap: Static analysis struggles with "implicit" interactions managed by frameworks (like Spring's
@PreAuthorizeannotations), which don't appear as explicit calls in the source code.
Key Results
The authors evaluated SecPatRec using four different LLMs (including GPT-6.1 and DeepSeek). Averaged across all models:
- Role Detection Accuracy (RDA): 0.84 (How well it identifies if a role exists).
- Mapping Localization Score (MLS): 0.84 (How accurately it pins a role to the correct line of code).
- Interaction Recovery Correctness (IRC): 0.66 (How accurately it traces the sequence of actions).
SecPatRec consistently outperformed a one-shot LLM baseline, particularly as the size of the codebase increased. While the baseline's performance plummeted once the code exceeded its token limit, SecPatRec’s accuracy remained stable.
Limitations and Future Work
The primary limitation identified is "framework blindness." Modern frameworks use dependency injection and annotations to link components at runtime. Because the CPG only sees explicit source code calls, it can miss these "invisible" links.
Future work aims to improve the CPG traversal by incorporating framework-specific configuration files and annotation parsing. Additionally, the authors noted that improving the initial role mapping is the highest-leverage way to increase overall recovery scores, as errors in the mapping phase naturally cascade into the interaction recovery phase.
Summary updated