TL;DR
AgentTrustCI is a Java/JUnit 5 framework designed to bring rigorous, build-time testing to LLM agents by evaluating their execution traces against predefined "behavioral contracts." By replaying recorded traces through a layered validator—checking tool policies, answer grounding, and call trajectories—it allows developers to catch logic errors and compliance violations before an agent ever hits production.
What Problem Does This Paper Address?
As LLM agents move from simple chatbots to autonomous workers capable of calling APIs (tools), they introduce new risks. Even if your infrastructure is secure, an agent might:
- Violate Policy: Call a
deletetool when it was only authorized tosearch. - Hallucinate (Lack of Grounding): Invent a ticket ID or price that wasn't in the tool's output.
- Take a Forbidden Path: Provide a correct answer but skip a mandatory approval step or access data through an unapproved sequence.
Traditional software testing often focuses on the final output, but for agents, the process is just as important as the result. Existing evaluation frameworks often rely on expensive, non-deterministic LLM-based "judges" or focus solely on runtime monitoring. AgentTrustCI fills the gap by providing a deterministic, fast, and repository-local way to run regression tests on agent behavior during the standard CI/CD build process.
Methods
The framework centers on the concept of a "Golden Case": a structured fixture containing a user request, a recorded execution trace (tool calls, parameters, and results), and a set of expectations.
The validation happens in three sequential layers:
- Policy Compliance: Ensures the agent only used allowed tools and provided all required (and no forbidden) parameters.
- Deterministic Grounding: Uses lexical matching to verify that identifiers, quantities, and quotes in the agent's final answer actually appeared in the tool results.
- Trajectory Validation: Checks the sequence of tool calls. It supports "strict" matching (exact sequence) and "relaxed" matching (ensuring required steps happen in order, even if other harmless calls are interleaved).
Crucially, the framework is designed for deterministic replay. It doesn't call a live LLM during the test; it evaluates how a validator would judge a stored trace, making tests fast and repeatable. It also includes a SafetyJudge extension point for teams that want to plug in LLM-based semantic evaluations for more complex reasoning.
Datasets
The authors evaluated the framework using a newly authored synthetic corpus:
- 55 Synthetic Cases: Spanning seven enterprise domains like support tickets, inventory, and audit logs.
- Composition: 30 cases with injected faults (10 per validation layer) and 25 passing controls (including 10 "tricky" cases designed to trigger false positives).
- Criticality: 12 of the fault cases were designated as "critical" (e.g., unauthorized data disclosure or administrative overrides).
This dataset is intended to test the coverage and separation of the validator layers rather than to serve as a general benchmark for agent intelligence.
Key Insights and Ideas
- The "Right-Answer-Wrong-Path" Trap: A major insight is that an agent can be factually correct but operationally dangerous. For example, an agent might correctly describe a product but get that info by bypassing a required catalog search and hitting a restricted database directly. Trajectory checks are essential to catch these workflow violations.
- Layered Defense: The authors found that while individual layers (Policy, Grounding, Trajectory) only caught about 63% of faults each (because the faults were isolated), the combined pipeline caught 100%. This highlights that no single check is a silver bullet for agent reliability.
- Developer-Centric Workflow: By using JUnit 5, the framework allows Java engineers to treat agent behavior like any other business logic. A failing agent behavior results in a failed build, providing immediate feedback in a familiar environment.
Key Results
- 100% Accuracy: On the 55-case test suite, the full pipeline correctly identified all 30 faults and accepted all 25 passing controls.
- Zero Missed Criticals: None of the 12 critical failures bypassed the validators.
- Extreme Efficiency: The validation is incredibly fast. The mean latency for the full three-layer pipeline was just 3.369 microseconds per case. The entire suite of 330 evaluations (running different modes) completes in less than 0.2 seconds, making it a negligible addition to build times.
Limitations and Future Work
- Lexical vs. Semantic: The current grounding check is lexical (string-based). It might miss a hallucination if the agent uses a synonym that isn't in the tool output, or it might miss a data leak if the sensitive value is disclosed without its field name.
- Linear Trajectories: The trajectory validator currently handles linear sequences. It does not yet support complex branching logic, concurrency, or data-dependent approvals.
- Synthetic Scope: The results are based on a small, hand-crafted corpus. The authors acknowledge that production environments will require a more robust protocol for capturing and labeling real-world traces to evolve these "golden cases."
- Future Directions: The authors suggest future work should focus on integrating calibrated LLM judges for semantic nuances and developing better tools for "promoting" production traces into the regression suite.
Summary updated