Introduction
As organizations rapidly integrate Large Language Models (LLMs) into their operational workflows, they often bypass rigorous security evaluations in favor of speed. While frameworks like the OWASP Top 10 for LLMs provide a taxonomy of potential threats, a significant gap remains: practitioners lack a standardized, actionable method to prioritize these risks within specific architectural contexts. This paper proposes a structured risk assessment framework that adapts the established OWASP Risk Rating Methodology to the unique challenges of AI-driven applications.
The Problem: Moving Beyond General Awareness
Existing security resources often fail to account for the diverse roles involved in LLM deployment. A prompt injection attack, for instance, carries vastly different implications for a developer building a simple API wrapper than for an engineer fine-tuning a foundational model on proprietary data. Without a method to quantify likelihood and impact across different stakeholder groups, LLM security remains a theoretical exercise rather than a manageable discipline.
Proposed Methodology
The authors introduce a semi-quantitative risk assessment process designed to move LLM security from "general awareness" to "actionable management." This process follows a three-step pipeline:
- Scenario Analysis: Security teams identify potential threat agents, evaluating their motives (e.g., financial gain vs. academic exploit) and the technical skill level required to execute an attack.
- Dependency Mapping: This step identifies which specific system components—such as prompt optimization modules, vector databases, or the LLM itself—are susceptible to vulnerabilities. By analyzing the system architecture, practitioners can estimate the likelihood of an attack based on factors like "Ease of Discovery" and "Public Awareness."
- Impact Analysis: The potential fallout is evaluated across two dimensions: Technical Impact (loss of confidentiality, integrity, or accountability) and Business Impact (financial damage, reputation loss, and privacy violations).
The final risk score is derived from the standard formula: Risk = Likelihood × Impact.
Stakeholder-Centric Risk Mapping
A core innovation of this framework is the mapping of threats to three distinct stakeholder groups: LLM Fine-tuning Developers, LLM API Integration Developers, and End Users. This granular approach acknowledges that security is not a monolith; different players have different "blast radiuses" and responsibilities.
Furthermore, the framework distinguishes between Static Controls (e.g., input validation, model obfuscation) and Dynamic Controls (e.g., real-time monitoring, adaptive trust boundaries), advocating for a layered defense strategy that addresses the "Dual Nature" of LLMs—where risks include both traditional cyber threats and AI-specific issues like "Excessive Agency."
Case Study: University Virtual Assistant
To demonstrate the framework, the authors applied it to a hypothetical University Virtual Assistant fine-tuned on administrative data and course materials. The assessment utilized the OWASP Top 10 for LLM Applications (v1.1.0) as its primary taxonomy.
Key findings from the case study include:
- Prompt Injection (HIGH Risk): Despite existing filters, the high "Ease of Discovery" and the potential for leaking student PII resulted in a high risk rating.
- Training Data Poisoning (MEDIUM Risk): While the technical impact of a poisoned model is severe, the likelihood was rated lower due to the difficulty of infiltrating a university's manual data vetting process compared to crafting a malicious prompt.
Conclusion and Future Work
While the proposed framework provides a robust template for risk assessment, the authors acknowledge that current ratings are based on qualitative expert judgment and hypothetical scenarios. Future work will focus on validating the framework through real-world industry case studies, refining the matrix as new "jailbreaking" techniques emerge, and developing automated tools to assist in the dependency mapping phase. By providing a structured way to quantify AI risks, this methodology enables organizations to embrace LLMs with a clear understanding of their security posture.
Summary updated