Manifest-Assisted Retrieval in Artificial Intelligence Search: Evidence Gaps and a Security Evaluation

Gangzhen Qian, Kainan Zhou, Chuhong Xu, Hang Xiao, Zhi Jing, Rong Bao, Lu Yi, Zhaoyi LiarXiv 2026 · arXiv

This paper evaluates the security and utility implications of the proposed /llms.txt manifest—a Markdown-based discovery aid designed to help AI agents navigate websites. The authors introduce a formal threat model and a "guarded consumption" framework, demonstrating that while raw manifests are highly vulnerable to indirect prompt injection, deterministic parsing can effectively neutralize these attacks without sacrificing utility.

TL;DR

The authors audit the emerging /llms.txt standard, finding that while current adoption and consumption are low, the format introduces significant risks for indirect prompt injection. They propose a "guarded consumption" strategy—stripping prose and using only validated links—which successfully blocked 100% of attacker-target adoptions in a pilot study while maintaining model performance.

What Problem Does This Paper Address?

As Retrieval-Augmented Generation (RAG) becomes the backbone of AI search, developers are looking for ways to help LLMs navigate the web more efficiently. The /llms.txt proposal (similar to robots.txt but for AI) provides a curated list of links and site overviews.

However, this creates a new "content-borne" attack surface. If an attacker can influence the prose or links within this manifest, they might execute an indirect prompt injection, hijacking the LLM's behavior (e.g., making it cite false information or perform unauthorized actions). The paper addresses the lack of systematic evidence regarding whether this new manifest actually improves utility or simply creates a dangerous security gap.

Methods

The researchers employed a multi-pronged approach:

  1. Evidence Audit: A non-systematic review of 39 sources to map the current landscape of AI search security and manifest adoption.
  2. Threat Modeling: They defined five specific "operational threat paths" (see Figure 1 and Table I) to categorize how an attack might manifest: Retrieval Abuse, Instruction Hijacking, Data Injection, Evidence Poisoning, and Ranking Manipulation.
  3. Guarded Consumption Framework: A proposed defense where the system treats manifest data as untrusted. It uses a deterministic parser to extract only same-origin HTTPS links, discarding any prose that could contain malicious instructions.
  4. Controlled Pilot Study: A 120-response experiment across eight real-world sites (like Stripe and Supabase). They tested five conditions, ranging from "absent manifest" to "raw adversarial manifest" and "guarded adversarial manifest," to measure security loss and utility.

![Fig.

  1. Proposed threat model. A content-borne attack crosses an exposure boundary and may follow one or more paths.](/figures/art_20503233897a4b7591f02f3e2e0cb4ef)

Datasets

  • Evidence Audit: 39 sources including research papers, standards (like robots.txt), and operational studies.
  • Pilot Study: Purposively selected documentation from 8 sites: WithJoy, Stripe, Supabase, ClickUp, Linear, PostHog, Zapier, and Asana.
  • Tasks: 24 researcher-authored tasks (3 per site) requiring factual answers based on the documentation.
  • Model: gpt-4.1-mini-2025-04-14 was used for the pilot generations.

Key Insights and Ideas

  • The Discovery-Consumption Gap: A key finding is that presence does not equal exposure. While some sites have adopted /llms.txt, 97% of those files received zero recorded requests in a large-scale log study. This suggests we are in a "pre-exposure" window where defenses can be built before widespread exploitation.
  • Pipeline Survival: Attacks often degrade as they move through the RAG pipeline (retrieval -> reranking -> generation). Figure 3 illustrates how cumulative filtering reduces attack success, though dense retrieval methods tend to be more "hospitable" to adversarial content than traditional BM25.
  • Prose is the Poison: In the pilot, almost all successful hijacks were driven by the prose descriptions in the manifest. By simply ignoring the prose and only following the links (the "guarded" approach), the researchers could reap the navigational benefits of the manifest without the injection risk.
  • Shadow Mode Deployment: The authors suggest that new AI features like manifest consumption should initially run in "shadow mode"—logging what would have happened without actually changing the user-facing output—to measure incremental utility vs. security risk.
Table I: Operational attack paths and outcomes.

Key Results

  • Vulnerability of Raw Manifests: In the "adversarial, raw" condition, the model adopted the attacker's false target in 21 out of 24 trials (87.5%–91.7% security loss).
  • Effectiveness of Guards: The "adversarial, guarded" condition (stripping prose) reduced known target adoptions to 0 out of 24.
  • Utility Stability: Using a manifest (even a guarded one) did not significantly harm utility. The "benign, guarded" condition resulted in 22/24 accepted answers, identical to the "absent manifest" baseline.
  • Latency: The mean request latency was low (0.628 seconds), suggesting that the overhead of manifest processing is negligible compared to the generation itself.

Limitations and Future Work

  • Scale: The pilot study was small (24 tasks, one model snapshot). Larger, more diverse benchmarks are needed to confirm these findings across different LLM architectures.
  • Static Evidence: The pilot used fixed evidence rather than live retrieval. Future work should evaluate how manifests affect the discovery phase in a dynamic web environment.
  • Adaptive Attacks: The study used fixed attack templates. The authors acknowledge that sophisticated, adaptive attackers might find ways to bypass deterministic guards (e.g., through malicious URL structures or metadata).
  • Human Adjudication: The pilot relied on AI-assisted scoring; independent human verification would strengthen the results.

Summary updated

Abstract

The external sources retained in our non-systematic audit did not isolate the incremental security or utility effect of consuming /llms.txt while ordinary retrieval remained fixed. The initial audit covered 39 sources through August 16, 2026, with later targeted updates through October 14. A purposive 15-tool study observed no automatic requests in 45 trials at one endpoint; a non-probability log study found that 97% of domains with detected files received no recorded requests in May 2026. We separate publication, fetch, model use, and action; define five operational threat paths and paired security-utility estimands; and exercise deterministic pre-model controls with 30 author-defined parser fixtures. We also execute a reduced pilot: 24 tasks across eight sites, five conditions, and 120 real model responses, with ordinary evidence held fixed. Frozen automatic scoring identifies attacker-target adoption in $21/24$ raw adversarial responses, with one further response unresolved; the adversarial URLs-only condition has no known adoptions and one unresolved response. The pilot measures output changes conditional on supplied evidence, without live retrieval or external actions. Together, the framework and pilot distinguish manifest-content effects from publication, discovery, and downstream execution, while leaving full retrieval and agent evaluation open.
BibTeX
@misc{qian2026manifest,
  author = {Gangzhen Qian and Kainan Zhou and Chuhong Xu and Hang Xiao and Zhi Jing and Rong Bao and Lu Yi and Zhaoyi Li},
  title = {{Manifest-Assisted Retrieval in Artificial Intelligence Search: Evidence Gaps and a Security Evaluation}},
  year = {2026},
  eprint = {2610.38361},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38361},
}
Figure