LLM-enabled Applications Require System-Level Threat Monitoring

Yedi Zhang, Haoyu Wang, Xianglin Yang, Jin Song Dong, Jun SunarXiv 2026 · arXiv

TL;DR

LLM-enabled applications introduce non-deterministic vulnerabilities that cannot be fully solved by pre-deployment testing, alignment, or localized guardrails. This position paper argues for an Endpoint Detection and Response (EDR)-inspired paradigm shift, treating security threats as expected operational conditions rather than exceptional events. To achieve this, the authors propose a comprehensive system-level monitoring schema that maps 14 threat categories to specific runtime artifacts and multi-stage audit logging pipelines.

What Problem Does This Paper Address?

Traditional software relies on predictable, symbolic logic, making it possible to define clear diagnostic signatures and enforce strict safety guarantees. In contrast, Large Language Model (LLM) applications are learning-driven and non-deterministic, creating a massive, fluid attack surface that includes prompt injection, data poisoning, and model drift.

Currently, the AI safety community focuses heavily on preventive measures: alignment techniques (like RLHF), pre-deployment red-teaming, and localized input/output guardrails. The authors argue that these defenses are fundamentally insufficient on their own. Guardrails lack global visibility across complex, multi-step workflows, and no non-trivial system can be expected to operate with complete immunity to failures. The critical gap addressed here is the lack of a systematic, system-level runtime monitoring and incident-response framework tailored to the unique operational characteristics of LLM applications.

Methods

The paper introduces a unified, system-level threat monitoring schema designed to capture anomalies across an application's entire lifecycle.

First, the authors establish a representative 8-stage agentic workflow utilizing the Model Context Protocol (MCP) to standardize interactions between the client, the LLM "brain," external tools, and retrieval-augmented generation (RAG) environments.

Figure 1. A respresentative LLM-enabled application workflow: The user submits an initial prompt (Stage 1); the client, responsible for orchestration, queries the MCP service for available tools (Stage 2) and forwards an integrated prompt to the LLM brain (Stage 3), which may interact with external resources such as vector databases, (Graph)-RAG systems, and memory (Stage 3*). The brain produces intermediate responses and tool plans (Stage 4); the client executes the selected tools via MCP and gathers results (Stage 5), assembles the final prompt (Stage 6), obtains the final response from the brain (Stage 7), and delivers it to the user (Stage 8).

Second, they systematically analyze 14 distinct threat classes. For each category, they map abstract attack vectors to concrete, measurable monitoring artifacts and outline how a multi-stage audit logging pipeline should ingest and analyze them. For example:

  • Lexical Obfuscation: Monitored via Unicode normalization diffs and unusual tokenization metrics (tokens-per-character ratios) to catch hidden or zero-width characters.
  • Embedding-level Attacks: Monitored via embedding outlier scores (Euclidean distance relative to historical benign queries) to detect deceptive semantic framing.
  • Environment Tampering: Monitored via memory mutation logs at the RAG/database layer to flag unauthorized write operations.

Datasets

As a position paper focusing on architectural and conceptual frameworks, it does not introduce any new datasets or benchmarks, nor does it conduct empirical training or evaluation.

Key Insights and Ideas

  • Vulnerabilities as Expected Operational Conditions: Instead of treating security breaches as rare anomalies, engineers must design systems under the assumption that failures will happen. The goal shifts from absolute prevention to rapid detection, containment, and forensics.
  • The Blind Spot of Localized Guardrails: Guardrails typically evaluate isolated inputs or outputs. They are blind to distributed, multi-turn exploits like contextual drift or prompt chaining, where an attacker uses a series of seemingly innocent prompts to slowly erode the model's safety alignment over time.
  • Recency Bias as a Structural Amplifier: In long-context scenarios (like instruction flooding via service APIs), malicious instructions appended near the end of a prompt are statistically more likely to be executed due to the model's inherent recency bias. Monitoring the insertion position of external data is therefore a critical system-level requirement.

Key Results

As a position paper, it does not present quantitative experimental results (such as accuracy or speedup metrics). Instead, its primary contribution is a comprehensive qualitative blueprint: a rigorous taxonomy mapping 14 threat categories to specific runtime monitoring artifacts and multi-stage audit logging strategies across the application lifecycle.

Table 1. The threat categories and associated attack vectors (or attack surfaces) for LLM-based applications considered in this paper.

Limitations and Future Work

The authors highlight three major challenges to implementing this monitoring schema in production:

  1. Semantic Ambiguity: Because natural language is inherently fluid, distinguishing between a complex, legitimate user instruction and a malicious override remains incredibly difficult, creating a tight trade-off between false positives and false negatives.
  2. Inspection Latency: Using secondary LLMs to inspect and evaluate runtime contexts for threats adds significant inference latency. Future work must focus on optimizing tiered strategies that combine lightweight, deterministic pre-filters with selective LLM analysis.
  3. Limited Observability in Closed-Source Models: Purely black-box API access prevents developers from auditing contextual provenance or internal state embeddings. The authors note that independent investigation is currently unfeasible without cloud service providers stepping up to offer standardized, controlled observability interfaces.

Summary updated

Abstract

LLM-enabled applications are rapidly reshaping the software ecosystem by using large language models as core reasoning components for complex task execution. This paradigm shift, however, introduces fundamentally new reliability challenges and significantly expands the security attack surface, due to the non-deterministic, learning-driven, and difficult-to-verify nature of LLM behavior. In light of these emerging and unavoidable safety challenges, we argue that such risks should be treated as expected operational conditions rather than exceptional events, necessitating a dedicated incident-response perspective. Consequently, the primary barrier to trustworthy deployment is not further improving model capability but establishing system-level threat monitoring mechanisms that can detect and contextualize security-relevant anomalies after deployment -- an aspect largely underexplored beyond testing or guardrail-based defenses. Accordingly, this position paper advocates systematic and comprehensive monitoring of security threats in LLM-enabled applications as a prerequisite for reliable operation and a foundation for dedicated incident-response frameworks.
BibTeX
@misc{zhang2026llm,
  author = {Yedi Zhang and Haoyu Wang and Xianglin Yang and Jin Song Dong and Jun Sun},
  title = {{LLM-enabled Applications Require System-Level Threat Monitoring}},
  year = {2026},
  eprint = {2602.19844},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.19844},
}
Figure