Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment Proposal

Rahul Pankajakshan, Sumitra Biswal, Yuvaraj Govindarajulu, Gilad GresselarXiv 2024 · DOI · arXiv

Introduction

As organizations rapidly integrate Large Language Models (LLMs) into their operational workflows, they often bypass rigorous security evaluations in favor of speed. While frameworks like the OWASP Top 10 for LLMs provide a taxonomy of potential threats, a significant gap remains: practitioners lack a standardized, actionable method to prioritize these risks within specific architectural contexts. This paper proposes a structured risk assessment framework that adapts the established OWASP Risk Rating Methodology to the unique challenges of AI-driven applications.

The Problem: Moving Beyond General Awareness

Existing security resources often fail to account for the diverse roles involved in LLM deployment. A prompt injection attack, for instance, carries vastly different implications for a developer building a simple API wrapper than for an engineer fine-tuning a foundational model on proprietary data. Without a method to quantify likelihood and impact across different stakeholder groups, LLM security remains a theoretical exercise rather than a manageable discipline.

Proposed Methodology

The authors introduce a semi-quantitative risk assessment process designed to move LLM security from "general awareness" to "actionable management." This process follows a three-step pipeline:

  1. Scenario Analysis: Security teams identify potential threat agents, evaluating their motives (e.g., financial gain vs. academic exploit) and the technical skill level required to execute an attack.
  2. Dependency Mapping: This step identifies which specific system components—such as prompt optimization modules, vector databases, or the LLM itself—are susceptible to vulnerabilities. By analyzing the system architecture, practitioners can estimate the likelihood of an attack based on factors like "Ease of Discovery" and "Public Awareness."
  3. Impact Analysis: The potential fallout is evaluated across two dimensions: Technical Impact (loss of confidentiality, integrity, or accountability) and Business Impact (financial damage, reputation loss, and privacy violations).
Figure 1: System architecture for the University Virtual Assistant case study, illustrating the components involved in the dependency mapping phase.

The final risk score is derived from the standard formula: Risk = Likelihood × Impact.

Stakeholder-Centric Risk Mapping

A core innovation of this framework is the mapping of threats to three distinct stakeholder groups: LLM Fine-tuning Developers, LLM API Integration Developers, and End Users. This granular approach acknowledges that security is not a monolith; different players have different "blast radiuses" and responsibilities.

Furthermore, the framework distinguishes between Static Controls (e.g., input validation, model obfuscation) and Dynamic Controls (e.g., real-time monitoring, adaptive trust boundaries), advocating for a layered defense strategy that addresses the "Dual Nature" of LLMs—where risks include both traditional cyber threats and AI-specific issues like "Excessive Agency."

Case Study: University Virtual Assistant

To demonstrate the framework, the authors applied it to a hypothetical University Virtual Assistant fine-tuned on administrative data and course materials. The assessment utilized the OWASP Top 10 for LLM Applications (v1.1.0) as its primary taxonomy.

Figure 2: The comprehensive Threat Matrix (Table I) mapping LLM risks to causes, consequences, and concerned stakeholders.

Key findings from the case study include:

  • Prompt Injection (HIGH Risk): Despite existing filters, the high "Ease of Discovery" and the potential for leaking student PII resulted in a high risk rating.
  • Training Data Poisoning (MEDIUM Risk): While the technical impact of a poisoned model is severe, the likelihood was rated lower due to the difficulty of infiltrating a university's manual data vetting process compared to crafting a malicious prompt.

Conclusion and Future Work

While the proposed framework provides a robust template for risk assessment, the authors acknowledge that current ratings are based on qualitative expert judgment and hypothetical scenarios. Future work will focus on validating the framework through real-world industry case studies, refining the matrix as new "jailbreaking" techniques emerge, and developing automated tools to assist in the dependency mapping phase. By providing a structured way to quantify AI risks, this methodology enables organizations to embrace LLMs with a clear understanding of their security posture.

Summary updated

Abstract

The rapid integration of Large Language Models (LLMs) across diverse sectors has marked a transformative era, showcasing remarkable capabilities in text generation and problem-solving tasks. However, this technological advancement is accompanied by significant risks and vulnerabilities. Despite ongoing security enhancements, attackers persistently exploit these weaknesses, casting doubts on the overall trustworthiness of LLMs. Compounding the issue, organisations are deploying LLM-integrated systems without understanding the severity of potential consequences. Existing studies by OWASP and MITRE offer a general overview of threats and vulnerabilities but lack a method for directly and succinctly analysing the risks for security practitioners, developers, and key decision-makers who are working with this novel technology. To address this gap, we propose a risk assessment process using tools like the OWASP risk rating methodology which is used for traditional systems. We conduct scenario analysis to identify potential threat agents and map the dependent system components against vulnerability factors. Through this analysis, we assess the likelihood of a cyberattack. Subsequently, we conduct a thorough impact analysis to derive a comprehensive threat matrix. We also map threats against three key stakeholder groups: developers engaged in model fine-tuning, application developers utilizing third-party APIs, and end users. The proposed threat matrix provides a holistic evaluation of LLM-related risks, enabling stakeholders to make informed decisions for effective mitigation strategies. Our outlined process serves as an actionable and comprehensive tool for security practitioners, offering insights for resource management and enhancing the overall system security.
BibTeX
@misc{pankajakshan2024mapping,
  author = {Rahul Pankajakshan and Sumitra Biswal and Yuvaraj Govindarajulu and Gilad Gressel},
  title = {{Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment Proposal}},
  year = {2024},
  eprint = {2403.13309},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.13309},
}
Figure