TL;DR
LLM agents often fail when APIs return "silent" errors—plausible but incorrect data delivered with a successful HTTP 200 status. This paper introduces SilentGuard, a system that moves verification logic from the LLM into the execution harness, using mathematical invariants to detect and block faulty tool outputs before the model ever sees them.
What Problem Does This Paper Address?
When an LLM agent calls an API, it assumes the result is correct if the call succeeds. However, APIs frequently suffer from silent failures. For example, if an agent sends a misnamed filter (e.g., organism=human instead of the documented human[orgn]), many production APIs will simply ignore the unrecognized parameter and return the unfiltered total count.
The agent receives a valid-looking number and a "200 OK" status, leading it to confidently report a hallucinated or irrelevant result. Previous attempts to fix this—like "LLM-as-a-judge" or self-correction checklists—rely on the model to catch the error. The authors discover a frustrating reality: LLMs are stubborn. Even when a secondary model flags a result as suspicious, agents often ignore the warning and report the faulty data anyway.
Methods
The authors propose SilentGuard, an "invariant-aware harness enforcement" system. Instead of asking the LLM to be skeptical, they build skepticism into the harness (the code that actually executes the API calls).
Key components include:
- Exact Invariants: These are properties that must be true for any correct API. For example, the "Count Identity": the count of items matching a filter $F$ plus the count of items matching $\neg F$ must equal the total count. If this doesn't add up, the API is objectively failing.
- Spec Checks: Automatically verifying that the agent's requested parameters exist in the API's documented schema.
- Learned Contracts: For cases where mathematical invariants don't exist, the system mines "contracts" (e.g., expected data types or value ranges) from clean historical data.
- Enforcement (The "Guard"): If a result fails an exact check, the harness withholds the value from the agent. The agent is told the result failed a consistency check and is prompted to either try a different approach or admit the answer cannot be determined.
Datasets
The study utilizes a mix of synthetic and real-world evaluations:
- AgentOrchBench (New): A benchmark created by the authors featuring 66 tasks (counting and listing) across eight local REST services. It allows for exact grading without needing an LLM judge.
- Real Public APIs: The authors probed 15 major research APIs (including NCBI, UniProt, and arXiv). They found that 11 of them exhibited silent failures, sometimes returning counts inflated by up to 691x.
- ToolMaze: An existing benchmark for multi-step tool workflows used to test how SilentGuard handles complex, interdependent tool calls.
Key Insights and Ideas
- Warnings are not enough: In tests, agents reported a flagged (incorrect) count in 77.4% of the runs where a judge explicitly warned them the data was suspect. Enforcement—actually blocking the data—is the only reliable way to prevent the error.
- The "Evidence-Permission-Enforcement" Triad: To stop failures, you need Evidence (a test showing the result is wrong), Permission (prompting the model that it is allowed to say "I don't know"), and Enforcement (the harness taking the decision away from the model).
- Metamorphic Testing at Runtime: SilentGuard essentially performs metamorphic testing (checking relationships between different inputs/outputs) in real-time during the agent's execution.
Key Results
SilentGuard demonstrated massive improvements in reliability across various model classes:
- Open-Weight Models: On AgentOrchBench, wrong answers dropped from 74.0% to 1.7%.
- Frontier Models (GPT-4/5 class): Wrong answers fell from 70.2% to 0.5%.
- Real-World Impact: On 9 public APIs with slightly incorrect documentation, SilentGuard cut wrong answers from 89.8% to 21.3%.
- Zero Overhead on Success: The system maintained high accuracy on "clean" tasks (84.9%), meaning the safety checks didn't break the agent's ability to function normally when tools worked correctly.
Limitations and Future Work
- Consistent Lies: If an API is "consistently" wrong (e.g., it lies about both the filtered count and the negated count in a way that satisfies the identity), SilentGuard cannot detect the failure. It requires an independent source of truth to catch "consistent" hallucinations.
- Query String Complexity: The current system struggles to inspect filters hidden inside complex, unstructured query strings (like those used in some legacy bioinformatics APIs).
- Abstention vs. Recovery: SilentGuard is currently better at abstaining (saying "I don't know") than recovering (fixing the call). Future work could involve the harness suggesting specific fixes to the agent based on why the invariant failed.
For more details, the code and AgentOrchBench dataset are available at github.com/Sajib-006/silentguard.
Summary updated