Trustworthy Agentic AI Requires Deterministic Architectural Boundaries

Manish Bhattarai, Minh Vu

Abstract

Current agentic AI architectures are fundamentally incompatible with the security and epistemological requirements of high-stakes scientific workflows. The problem is not inadequate alignment or insufficient guardrails, it is architectural: autoregressive language models process all tokens uniformly, making deterministic command–data separation unattainable through training alone. We argue that deterministic, architectural enforcement, not probabilistic learned behavior, is a necessary condition for trustworthy AI-assisted science. We introduce the Trinity Defense Architecture, which enforces security through three mechanisms: action governance via a finite action calculus with reference-monitor enforcement, information-flow control via mandatory access labels preventing cross-scope leakage, and privilege separation isolating perception from execution. We show that without unforgeable provenance and deterministic mediation, the "Lethal Trifecta" (untrusted inputs, privileged data access, external action capability) turns authorization security into an exploit-discovery problem. The ML community must recognize that alignment is insufficient for authorization security, and that architectural mediation is required before agentic AI can be safely deployed in consequential scientific domains.

Introduction: The Failure of Learned Separation

As we transition toward "Agentic AI"—systems capable of browsing the web, utilizing tools, and managing persistent memory—we are inadvertently recreating classic security vulnerabilities such as SQL injection. The core issue lies in uniform token processing.

Because autoregressive transformers process all input tokens through the same attention mechanism, they lack an architectural method to verify the provenance (origin) of a command. Whether a token originates from a trusted developer or an untrusted external document, the model treats it with equal weight. This allows for "multi-modal bypasses" where malicious instructions hidden in PDFs or images can steer the agent's behavior. Training-based defenses are content-based and therefore forgeable; an attacker can always find a semantic representation that bypasses a learned filter.

Figure 1: Learned separation fails because LLMs process trusted instructions and untrusted data tokens uniformly, providing no unforgeable provenance.

The Lethal Trifecta

The risk is most acute in scientific workflows where three conditions—the Lethal Trifecta—often hold simultaneously:

  1. Untrusted Inputs: The agent ingests external research papers or web data.
  2. Privileged Data Access: The agent has access to internal lab results, proprietary datasets, or credentials.
  3. External Action Capability: The agent can execute tool calls, such as sending emails, modifying databases, or triggering laboratory hardware.

When these three conditions intersect, probabilistic compliance (alignment) does not equal security. An injected instruction can steer tool use toward exfiltration or unauthorized actions, creating a critical failure point for scientific integrity.

The Trinity Defense Architecture

To solve this, we propose moving security logic out of the probabilistic LLM and into a deterministic Trusted Control Plane. The Trinity Defense Architecture relies on three pillars to enforce deterministic boundaries:

Figure 2: The Trinity Defense Architecture enforces security through deterministic action governance, information-flow control, and privilege separation.

1. Action Governance (Finite Action Calculus)

Instead of allowing the LLM to generate arbitrary tool calls, the system employs a Finite Action Calculus (FAC). A non-LLM Reference Monitor (Command Gate) intercepts every proposed action, checking it against a strict, predefined policy. If an action is not explicitly authorized, it is blocked at a single choke point before execution.

2. Information-Flow Control (IFC)

Data is assigned mandatory security labels (e.g., UNTRUSTED, INTERNAL, CONFIDENTIAL). Using a lattice-based model, the system ensures that data from a CONFIDENTIAL source cannot flow into an UNTRUSTED sink (like an outgoing email) without an explicit, audited declassification step. This prevents the LLM from accidentally or maliciously leaking sensitive data.

3. Privilege Separation

The architecture bifurcates the agent into a Planner and a Worker. The Planner (low privilege) interacts with untrusted content to formulate strategies but lacks the authority to execute tools. The Worker (high privilege) executes authorized actions but only receives sanitized, normalized inputs from the Planner, never raw untrusted data.

Theoretical Foundations and Key Insights

Our analysis yields several critical insights for the future of AI security:

  • Alignment is not Security: We demonstrate that no amount of training can provide a deterministic guarantee of command-data separation if both are processed in the same token stream. A "perfectly aligned" model that prioritizes helpfulness will, by definition, follow a malicious instruction if it appears helpful.
  • The Epistemic Crisis: In scientific research, the "chain of evidence" is paramount. Without architectural provenance, we cannot know if an agent's conclusion was derived from peer-reviewed data or an adversary's injected hallucination.
  • Security through Architecture: We must treat the LLM as an untrusted component. Rather than attempting to make the LLM "smarter" to avoid attacks, we should build a control plane that makes unauthorized actions architecturally impossible.

Conclusion

Trustworthy AI in science requires more than better models; it requires a paradigm shift toward Agentic Good Laboratory Practice (aGLP). By implementing deterministic boundaries through the Trinity Defense Architecture, we can ensure that agentic systems remain helpful assistants rather than liabilities. Future work must focus on minimizing authorization latency and solving the challenge of "label creep" to ensure these systems remain both secure and performant.

Summary updated

Abstract

Current agentic AI architectures are fundamentally incompatible with the security and epistemological requirements of high-stakes scientific workflows. The problem is not inadequate alignment or insufficient guardrails, it is architectural: autoregressive language models process all tokens uniformly, making deterministic command--data separation unattainable through training alone. We argue that deterministic, architectural enforcement, not probabilistic learned behavior, is a necessary condition for trustworthy AI-assisted science. We introduce the Trinity Defense Architecture, which enforces security through three mechanisms: action governance via a finite action calculus with reference-monitor enforcement, information-flow control via mandatory access labels preventing cross-scope leakage, and privilege separation isolating perception from execution. We show that without unforgeable provenance and deterministic mediation, the ``Lethal Trifecta'' (untrusted inputs, privileged data access, external action capability) turns authorization security into an exploit-discovery problem: training-based defenses may reduce empirical attack rates but cannot provide deterministic guarantees. The ML community must recognize that alignment is insufficient for authorization security, and that architectural mediation is required before agentic AI can be safely deployed in consequential scientific domains.
BibTeX
@misc{bhattarai2026trustworthy,
  author = {Manish Bhattarai and Minh Vu},
  title = {{Trustworthy Agentic AI Requires Deterministic Architectural Boundaries}},
  year = {2026},
  eprint = {2602.09947},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.09947},
}
Figure