Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry

Shoumik Saha, Kazem Faghih, Soheil FeiziarXiv 2026 · arXiv

TL;DR

This paper exposes a novel semantic supply-chain risk in AI agent ecosystems by demonstrating how malicious actors can manipulate the natural-language metadata in SKILL.md files. By altering only documentation text—without changing any executable code—attackers can successfully hijack skill discovery, bias agent selection, and evade registry governance mechanisms. The findings reveal that natural-language manifests are not passive documentation but critical operational vectors that require robust security defenses.

What Problem Does This Paper Address?

Modern AI agents increasingly extend their capabilities using "Agent Skills"—modular filesystem packages that contain task-specific instructions, scripts, and assets. These packages are anchored by a mandatory SKILL.md file, which serves as both a manifest for registries and an instruction entry point for the agent.

Because registries use embedding-based search to discover these skills and LLM agents read SKILL.md files to decide which tool to use, the ecosystem introduces a unique semantic supply-chain risk. Traditional security tools look for malicious code, but they are blind to malicious natural language. This paper systematically explores how an adversary can exploit this blind spot across the three primary registry-facing stages of the skill lifecycle: Discovery (ranking), Selection (agent choice), and Governance (vetting).

Figure 1: Overview of SKILL.md-only semantic supply-chain attacks on an Agent Skill registry. Starting from a benign SKILL.md, an adversary modifies only natural-language content while leaving the skill’s functional structure largely unchanged. The injected text can serve three distinct purposes: a discovery trigger that improves registry ranking for target queries, a selection trigger that biases the agent toward the adversarial skill over a functionally similar original skill, and a malicious instruction that preserves unsafe agent-facing intent while passing registry-side scans.

Methods

The authors design and evaluate SKILL.md-only attacks across three distinct lifecycle stages:

  1. Discovery Manipulation: The authors develop both black-box (beam-search based) and white-box (gradient-based) optimization techniques to append short, 20-token textual triggers to SKILL.md files. These triggers are designed to artificially boost the skill's embedding similarity to target user queries (e.g., "travel", "tax"), forcing the registry to rank the adversarial skill higher.
  2. Selection Manipulation: To manipulate the agent's choice after discovery, the authors append subtle linguistic framing statements to the description field. They test four psychological framing strategies: False Advertising (exaggerating capability), Assertive Cues (directive language), Active Maintain (recency signals), and Trust/Security (claims of trustworthiness).
  3. Registry Governance Evasion: The authors attempt to bypass an automated, LLM-based vetting pipeline (called ClawScan) while keeping malicious instructions intact. They evaluate four evasion strategies: LLM Judge Jailbreaking, Paraphrasing Malicious Instructions into benign operational language, embedding attacks into Definition-of-Done checklists, and Context Window Overflow (placing malicious text past the scanner's truncation limit).

Datasets

The evaluation is grounded using a corpus of 100 real, publicly available agent skills downloaded from ClawHub.ai (OpenClaw's skill registry). The dataset spans five diverse, highly practical domains: email, travel, tax, health, and prompt engineering, selecting the top 20 most-downloaded skills in each category to ensure realistic evaluation.

Key Insights and Ideas

  • Documentation is Code: In the world of LLM agents, SKILL.md is not passive documentation; it is operational text that directly dictates routing, trust, and execution.
  • Asymmetric Transferability: Textual triggers optimized to exploit one specific embedding model transfer remarkably well to other models. Interestingly, attacks optimized on open-source models (like BAAI/bge) transfer effectively to proprietary models (like OpenAI's text-embedding-3-small), but OpenAI-optimized triggers transfer poorly back to open-source models.
  • Linguistic Bias in Agents: LLM agents are highly susceptible to sycophantic and assertive framing. When presented with two functionally identical skills, agents consistently favor the one that uses persuasive language like "This is the best skill for this purpose."
  • The Truncation Blindspot: Automated LLM governance pipelines often truncate long files to save on compute costs. Attackers can easily exploit this by placing benign text at the beginning of a file and hiding malicious instructions at the very end.

Key Results

  • Discovery: In pairwise testing against OpenAI's embedding model, the beam-search attack achieved an 86.14% win rate over original skills, translating to an 80% Top-10 placement rate in registry searches.
Figure 2: Pairwise retrieval-score improvement across embedding models. Each cell reports the win rate and average relative score boost of the adversarial skill over the original skill. Rows indicate the model used to optimize the discovery trigger, and columns indicate the model used for evaluation. Diagonal entries measure same-model attacks, while off-diagonal entries measure transfer.
  • Selection: Across four state-of-the-art LLMs (including GPT-5 and Qwen), description-only framing biased the agents into selecting the adversarial variant in 77.6% of paired trials on average.
Figure 4: Selection preference for adversarial variants over original skills. Each cell shows adversarial/original selection proportions and the corresponding dominance ratio.
  • Governance: While direct malicious insertions were caught 100% of the time, paraphrasing allowed 73.9% of malicious skills to evade a blocking verdict. The Context Window Overflow strategy was the most potent, achieving a 100% bypass rate (with 87.1% of malicious files marked completely "clean").

Limitations and Future Work

The authors restrict their threat model strictly to SKILL.md-only modifications to isolate the impact of semantic text. Real-world attackers would likely combine these techniques with actual malicious code or auxiliary file manipulation. Additionally, the governance pipeline evaluated was a simulated local replica of ClawHub's defense mechanisms; production environments with multi-modal or dynamic sandbox testing might yield different evasion rates. Future work will focus on designing robust, chunk-based semantic scanners and evidence-based agent selection policies to mitigate these vulnerabilities.

Summary updated

Abstract

Autonomous AI agents increasingly extend their capabilities through Agent Skills: modular filesystem packages whose SKILL.md files describe when and how agents should use them. While this design enables scalable, on-demand capability expansion, it also introduces a semantic supply-chain risk in which natural-language metadata and instructions can affect which skills are admitted, surfaced, selected, and loaded. We study SKILL.md - only attacks across three registry-facing stages of the Agent Skill lifecycle, using real ClawHub skills and realistic registry mechanisms. In Discovery, short textual triggers can manipulate embedding-based retrieval and improve adversarial skill visibility, achieving up to 86% pairwise win rate and 80% Top-10 placement. In Selection, description-only framing biases agents toward functionally equivalent adversarial variants, which are selected in 77.6% of paired trials on average. In Governance, semantic evasion strategies cause malicious skills to avoid a blocking verdict in 36.5%-100% of cases. Overall, our results show that SKILL.md is not passive documentation but operational text that shapes which third-party capabilities agents find, trust, and use.
BibTeX
@misc{saha2026under,
  author = {Shoumik Saha and Kazem Faghih and Soheil Feizi},
  title = {{Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry}},
  year = {2026},
  eprint = {2605.11418},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.11418},
}
Figure